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SUMMARY 



The main purpose of National Assessment is to measure change 
in what children and young adults know and can do. Science 
was assessed for a national sample of respondents in 1969-70. 
Science will be assessed again in 1972-73. After that, reports 
of science results will emphasize what seems to have been 
learned better, and what less well. In the meantime, our reports 
are confined to the results of the only administration of science 
exercises conducted so far. 

The first step towards providing information about knowledge of 
Science on the part of American children and young adults was 
National Assessment Report 1, which gave national percentages of 
success for science exercises. In Report 4, results were 
examined to detect differences in percentages correct between the 
two sexes, among four geographical regions and among four sizes 
of community. 

The present report extends the results to cover the performance 
of Blacks, and of respondents with differing levels of parental 
education and from differing types of community. For each sub- 
group on each of these characteristics, we describe performance 
differences as they are . In addition, for each subgroup, in- 
cluding those examined in Report 4, we discuss differences as we 
estimate they would be if the effects of other characteristics 
were represented proportionately in each subgroup . 

The major results are: 



Performance of Blacks 



Blacks performed between 12% and 16% below the national 
average at the four age levels: 9, 13, 17 and young Adults 

(26-35). 

When results were partially adjusted ("balanced") for dis- 
porportionate representation of Blacks on the variables of size 
of community, level of parental education, sex and region, the 
reduced difference between Black and national performance was 
between -7% and -10% at the four age levels. 

Blacks performed best on those science exercises 
largely dependent upon daily experience and common knowledge, 
and worst on those which involved a detached research attitude 
toward the objects and phenomena of Science. 



Levels of Parental Education 



At the four ages performance of respondents reporting 
neither parent educated beyond eighth grade was from -7% to -12% 
compared to the national average; of respondents with at least 
one parent who attended high school, from -2% to -8% compared to 
the national average; with at least one parent who graduated from 
high school, from 1% below to 3% above national average; with at 
least one parent educated beyond high schook, from 5% to 9% 
above national average. 

Balancing for disproportionate representation of other 
variables somewhat reduces the effect of the parental education 
classification, although there is still about an 11% performance 
difference between the group of respondents who reported a 
parent educated beyond high school and the group of respondents 
who reported that neither parent attended high school. 

Respondents with neither parent completing high school had 
exceptional difficulty on exercises requiring special vocabulary 
knowledge or reasoning from graphs and pictures. 



Types of Community 

Respondents living in Extreme Affluent Suburbs did 5% to 11% 
better than the country as a whole, followed by four types of 
communities with performance within 1, 2, or 3% of national 
average — Suburban Fringe, Medium Cities, Small Cities and Inner 
City Fringe. Next in order were Extreme Rural respondents, from 
4 % to 6 % below the nation, and finally the Extreme Inner City 
with from 7% to 15% deficit. 

The balancing adjustment reduces the type of community effects 
to about half of their unadjusted size. 

Region 

The most striking difference between unadjusted and balanced 
regional effects is the reduction of the Southeast deficit of 
about 5% to about 2.5% . Balanced results for the other three 
regions cluster tightly about 1% above national average. 

Objectives 

As in National Assessment Report 4, there is a slight 
tendency for exercises assigned to Objective 2 (Possess the 
abilities and skills needed to engage in the processess of science) 
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to display stronger differences between groups than exercises 
assigned to Objective 1 (Know the fundamental facts and prin- 
ciples of Science) . Among the variables newly analyzed in this 
report, this tendency is most clear for color results. 

Limitations 



In the previous report of group results on Science (National 
Assessment Report 4) the following cautionary statement was 
made : 

There is a kind of interpretation that should never be made 
on the basis of the sort of figures given in this report. 

The fact that figures reflect Southeast performance or Big 
City performance does not mean that the performances thus 
reflected have arisen precisely from living in the South- 
east or in a Big City, or from the attitudes, techniques, 
facilities and staffs of the school system involved. In 
particular, just what happens in a region involves other 
things than that region's schools. Larger fractions of the 
children in some regions belong to a particular size-of- 
community group. Thus effects due only to size of community 
can appear to be regional differences. Larger or smaller 
fractions of the parents in some regions have particular 
amounts of education. Thus effects due only to parental ed- 
ucation can appear to be regional differences. And so on. 
Migration from one region or size of community to another 
can further complicate the picture. There are such diffi- 
culties, some of which we know how to adjust for, and some 
of which we do not. 

Great caution should also be exercised in interpreting the re- 
sults for the group variables from the present report. One 
should refrain from attempts to assign causation to any 
particular set of educational factors, or any variables from 
the heredity-environment complex, based on results reported here. 

We have attempted to adjust group results for disproportionate 
representation of other characteristics within each group. 

This "balancing" procedure is helpful, but it cannot adjust for 
variables which were not measured or not used in the analysis. 

Nor can it overcome either inaccuracy or coarseness in the 
classification of respondents or the fact that the variables 
that were measured may supply only an indirect indication of 
what we really would like to measure. 

Two other special limitations should be noted, as they apparently 
contribute to some irregularity in the patterns of results for 
age 17. The out-of-school sample included at this age suffers 
from the problem that many teen-agers are difficult to locate 



at home. Among the subgroup of 17s who cannot readily be found 
are probably a large number of individuals who have been quite 
unsuccessful in school and are antagonistic to schQlastic tasks. 
Thus the 17-year-olds available in the sample might be expected 
to perform well compared to unavailable 17-year-olds. The 
problem is compounded by the possibility that certain subgroups 
might be unavailable in greater proportion than other subgroups. 
This may be the case with 17-year-old Blacks, for example, since 
the number of Blacks in the 17-year-old sample is only about 3/4 
the number at ages 9 and 13. Thus, the 17-year-old Black sample 
might be selectively different from the Black samples at other 
ages. In fact, the results to be examined in Chapter 2 show 
that Black 17s perform relatively better than Blacks at other 
ages . 

The second irregularity at age 17 is that the distribution of 
exercise difficulties happens to have come out more heavily 
weighted toward very difficult exercises. (See page 10 of Re- 
port 1). To the extent that very difficult exercises differen- 
tiate subgroups less sharply than other exercises, the 
differences ®mong subgroups will be reduced at age 17. 
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Chapter 1 

WHAT IS BALANCING? 



This is the first of the NAEP reports to consider effects 
associated simultaneously with several classifications of 
respondents. Unfortunately, in a direct comparison between 
group effects, one characteristic can masquerade as effects of 
another. In addition to direct comparisons, we here use a 
method of analysis designed to reduce such masquerading. 

The unadjusted results as reported here and in Report 4 clearly 
and accurately estimate the differences in achievement between 
specific groups of children. For example, over all the science 
exercises, the median percentage difference between 13-year-olds 
in the Extreme Affluent Suburbs and in the Extreme Inner City 
is 20% (from Exhibit 6-1) . Except for sampling error, 
this accurately reflects how these two groups differ.* 

However, children in the Extreme Affluent Suburb tend, more 
than children in the Extreme Inner City, to have better educated 
parents. Because of this lack of balance, pari of the difference 
between these two groups may be considered as growing out of the 
difference in parental education. Part, also, may be attri- 
butable to other factors on which the two groups differ. Some 
of these factors have been determined for our respondents their 
sex, color and the region of residence. Many other possibly 
relevant factors have not been determined, such as the economic 
level of the children's parents and the cultural environment in 
the home . 

It is natural to ask, "What would the difference between these 
extreme types of community have been if the distribution of 
Parental Education, sex, color and region had been the same 
for both types of community referred to above?" Were it 
possible to rearrange the world to equate these distributions for 
each type of community, the effects upon our nation and its 
schools would be profound. Such rearrangement is not possible. 

It is usually appropriate to think of the balanced results pre- 
sented in this report as reflecting the differences we would see 
in the absence of masquerading by the other four factors. We 
can be reasonably sure the balanced results do a much better job 
than the unadjusted results of reflecting such differences. 



*The reader is referred to Appendix A for the description of 
the methods used for developing such comparisons . 



While balancing helps to correct for disproportionate repre- 
sentation across categories, it cannot adjust for variables 
which were not measured or not used in the analysis. Nor can 
it overcome either inaccuracy or coarseness in the classifi- 
cation of respondents or the fact that the variables that were 
measured may supply only an indirect indication of what we 
really would like to measure. 

Still another question concerns the combination of factors. 

The performance of a given subgroup may be found to differ, 
depending upon subgroupings on other variables. Thus, the 
effect associated with Extreme Affluent Suburbs may be different 
in the Northeast and the Southeast. Or the effect associated 
with sex may be somewhat different for Blacks and Non-Blacks . 
These interactive differences are not considered in this re- 
port and balancing does not adjust for them. 

Appendix B presents more details about the nature and com- 
putational procedures of balancing. 



Chapter 2 

COMPARATIVE PERFORMANCE OF BLACKS 



In view of the widespread national concern with the educational 
disadvantage of Blacks and other minority groups, National 
Assessment was designed so as to begin an evaluation of the 
details of such possible disadvantage. The color of each 
respondent was noted by the exercise administrator on 
the Assessment package as answers were turned in. During 
year 01, color w^s noted as "Black" or "Non-Black" by the 
administrator. The Non-Black category in addition to many 
whites, included Orientals, Mexican-Americans , Indians and 
some Puerto Ricans. The present chapters give results for 
Blacks in this first cycle of Science assessment. The 
standard measure of relative performance, on each exercise 
is the difference between the percentage of correct answers 
given by Blacks and the national percentage of correct 
answers. These measures give benchmarks for evaluating 
potential progress in later assessment cycles. 

The performance of the Black respondents in National 
Assessment can reflect individual, school or environmental 
factors, and there is no sure way to disentangle the complex 
web that represents the Black educational condition in 
America today. In the next chapter, a preliminary attempt 
is made to adjust Black results to take account of the 
general advantage for whites of disproportionate represen- 
tation in Affluent Suburbs, higher numbers of highly 
educated parents and so on. 

The effects for Blacks and Non-Blacks differ widely from 
exercise to exercise, as noted in National Assessment 
Report 1 (pp. 151, 155). In this and the next chapter we 
discuss individual exercises and types of exercises on 
which Black respondents perform atypically better or worse 
than their general level. 

Exhibit 2-1 shows the distribution of relative performances 
by Blacks over all exercises at the four age levels. The 
median results for Blacks are consistently below national 
results^ by 14.5%, 15.0%, 11.8% and 15.8% at ages 

9, 13, 17 and young Adults (26-35) . For individual 
exercises, the spread around these medians is considerable , 
however, as the exhibit shows. The smaller median deficit 
at age 17 than at the other three ages is difficult to 
interpret. (See Summary, p. iii.) In any case, it seems 
safe to say that there is no systematic decline in 
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Exhibit 2-1 



Distributions of relative performances of Blacks 
for all science exercises at four aqes 



Age 



9 




17 




4J H 

O 03 




-40 



No. of 

Exercises 145 122 124 



10 




0 



-10 



-20 



-30 



-40 



119 



Median -14.5% 



-15.0% 



- 11 . 8 % 



-15.8% 
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relative Black performance with increasing age. (This 
conclusion also applies if attention is focused solely 
on exercises administered at more than one age level.) 

Median relative performances by Blacks on exercises for 
each science objective are shown in Exhibit 2-2. Objectives 1 

and 2— covered -different forms of science achievement- with 

sizeable numbers of exercises. Objectives 3 and 4 touched 
upon attitudes and orientations toward Science and scientists, 
employing rather few exercises. Comparing Objective 1 
(Know the fundamental facts of Science) with Objective 2 
(Possess the abilities and skills to engage in the processes 
of Science) we note that at all four ages the Black 
disadvantage is consistently less for Objective 1,* 
by amounts of 2.2%, 3.7%, 8.7% and 3.0%. That is, Black 
respondents at all assessed age levels tend to perform 
relatively better on science fact exercises than on science 
process exercises. 

At ages 13 and 17, there is a tendency for the median Black 
deficit to be rather small (4 to 6%) for the exercises of 
Objective 4 (Show appreciation of scientists and Science) . 

At age 17, the deficit is also small (4%) on Objective 3 
(Understand the investigative nature of Science) • There 
are relatively few exercises under these two objectives, 
but the hint is nevertheless clear that whatever their 
achievement levels, Black teen-agers do not necessarily 
express a lack of appreciation for Science. 

Median results for physical science vs. biological science 
exercises are presented in Exhibit 2-3. There are no 
systematic differences in Black performance on these 
two types of science exercises. 



♦Region, sex and size-of-community groups also are less 
strongly differentiated on Objective 1 than Objective 2 
exercises. See Report 4, pp. 10, 5, 21. 
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Exhibit 2-2 



Median relative performance by Blacks for 
exercises under each objective 

Science 



Objective 


9 


13 


17 


Adult 


1 . 


Facts and 


-15.5% 


-15.0% 


-10.9% 


-14.8% 




Principles 


(96)* 


(75) 


(89) 


(85) 


2. 


Abilities and 


-17.7% 


-18.7% 


-19.6% 


-17.8% 




Skills 


(29) 


(31) 


(24) 


(24) 


3. 


Understand 

Investigative 


-12.1% 


-18.8% 


-4.2% 


-23.7% 




Nature 


(11) 


(8) 


(6) 


(5) 


4. 


Attitudes and 


-10.7% 


-4.3% 


-6.0% 


-10.6% 




Appreciations 


(9) 


(8) 


(5) 


(5) 



♦Numbers of exercises in parentheses 





Exhibit 2-3 



Median relative performance by Blacks for 
physical and biological science exercises 





9 


13_ 


17 


Adult 


Physical 

science 


-16.0% 

(78)* 


-15.0% 

(67) 


-12.7% 

(79) 


-16 . 6% 
(63) 


Biological 

science 


-14.2% 

(48) 


-15.4% 

(39) 


-10.8% 

(32) 


-16.7% 

(44) 



*Numbers of exercises in parentheses 

Note: Minus percent figures represent median percentages 

below national averages. 



Chapter 3 



COMPARATIVE PERFORMANCE OF BLACKS, BALANCED, 
FOR OTHER MEASURED FACTORS 



Blacks are subject to many adverse conditions, and it is 
extremely difficult to identify the factors most responsible 
for the comparative disadvantage in Science performance 
reported in the previous chapter. We can attempt to adjust 
our data for some of these factors through the application of 
"balancing" as explained in Chapter 1, and Appendix B. Many 
factors are not available in National Assessment data and we 
cannot, of course, adjust for these. Thus, the adjustments 
may well be too small, and therefore the balanced figures 
cannot be expected to represent the full potential in Science 
of Black children and young Adults under appropriate educa- 
tional stimulation at home and in school. 

Blacks are underrepresented in the most educationally advan- 
taged type of community — the affluent suburbs--and are over- 
represented in the educationally disadvantaged inner city. 
(Results for different types of communities, showing the 
advantages and disadvantages, are presented in Chapters 6 and 
7.) One might ask how much of the Black disadvantage could 
be accounted for in terms of this disproportionate represen- 
tation. Similarly, there are disproportionately few Blacks 
whose parents have had the benefits of Completed High School 
and Post- High School education (see Chapters 4 and 5) , and 
one might again ask how much the Black disadvantage would be 
reduced were Black and white levels of parental education 
balanced. A further factor involving disproportionate repre- 
sentation of Blacks is that of Region (see Chapter 8) . 

Exhibit 3-1 shows the distribution of relative performances 
by Blacks at all four age levels , balanced for disproportionate 
type-of- community , parental education, sex and reglonaT 
representation . (For the conceptual basis of the balancing 
operation, the interested reader is referred to Chapter 1, and 
for the arithmetic details, to Appendix B.) Following the 
balancing adjustment, the median results for. Blacks are still 
below national results, but by amounts that are substantially 
less than those which do not take the balancing factors into 
account. The residual disadvantages are -10.2%, -11.0%, -7.7% 
and -10.9% at ages 9, 13, 17 and young Adult, respectively. 
(Compared to -14.5%, -15.0%, -11.8% and -15.8% as noted before 
in Exhibit 2-1.) 



Exhibit 3-1 



Distributions of relative performances of Blacks, balanced, 
for all science exercises at four ages 
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The 17-year-old age group retains the smaller disadvantage 
noted in the previous chapter. Other patterns over exercises , 
such as the difference between Objective 1 and Objective 2 
exercises remain in force after balancing. There is 
a general reduction in the Black disadvantage, age by age, and 
exercise by exercise. Exhibit 3-2 displays the relationship 
over. . exercises between unad j usted and balanced results for 
each age. It is clear in general that on the exercises for 
which Blacks perform well before balancing, they also perform 
well after balancing. A similar statement could be made for 
exercises with relatively poor Black performance. 



Black % correct less national % correct 



Exhibit 3-2 

Relationship between '^adjusted and 
balanced results for all exercises 
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Atypical Exercises 



The procedure for identifying specific exercises on which 
Black performance (after balancing) is atypically high or 
atypically low is identical to the procedure used in National 
Assessment Report 4 for sex, region, and size-of-community 
groups. On the distributions of relative performances for all 
exercises, upper and lower cutoff values are placed (see 
arrows on Exhibit 3-1) in order to select and discuss exercises 
for which the Black disadvantage disappears, on the one hand, 
or is much worse than typical, on the other hand. The details 
of the procedure for selecting cutoffs are given in Appendix C. 



9-Year-Olds 



The exercises for which Blacks perform atypically at age 9 are 
listed in Exhibit 3-3. Exercises labeled "R" have beun released 
to the public (see National Assessment Report 1) and therefore 
their content may be discussed in detail. Exercises labled "U," 
however, are being retained for reuse in later assessment cycles 
and therefore are not published at this time. In Exhibit 3-3 
and throughout this report, no summary phrase in the "content" 
column is given for these unreleased exercises. 

In the upper section of Exhibit 3-3 are listed exercises on which 
Blacks performed atypically well. Most of these exercises proved 
difficult for the national sample as a whole. Thirteen of the 
sixteen exercises show national percent correct of 36% or below, 
with seven of these thirteen having correct percentages below 
20%. The 20% figure is what one would anticipate for a multiple- 
choice exercise answered by guessing at random among five 
alternatives , and therefore one may characterize national per- 
formance on these seven exercises as even worse than what 
respondents could obtain by guessing. (The situation actually 
is more complicated, since some percentage of respondents 
answered each exercise with "I don't know.") Altogether there 
are eight exercises at age 9 with national percentages below 
20%, of which seven appear in Exhibit 3-3. An eighth one shows 
a Black difference of only -3%. These very difficult exer- 
cises do not turn out to show substantial differences between 
groups that differ on most other exercises. 



Exhibit 3-3 



Exercises showing atypical effects, balanced, 
for Blacks for age 9 



% Correct 

Exer- % Dif- 

cise # Black Nat'l ference Content 



U652* 


35 


33 


2 




R141* 


9 


7 


2 


Mixing water of 50° and 70° yields 
water of 60° 


U673* 


12 


11 


1 




U650* 


37 


36 


1 




U655* 


15 


14 


1 




U678* 


27 


26 


1 




U657* 


12 


11 


1 




U621* 


76 


76 


0 




U656* 


12 


12 


0 




R140* 


14 


14 


0 


Dead plants form coal 


R142* 


96 


96 


0 


Balancing a beam with one weight 


U617 


78 


78 


0 




U677* 


33 


33 


0 




R158* 


22 


23 


“1 


What is a scientific theory 


U671 


39 


31 


-1 




U672* 


18 


19 


“1 




U6l2 T 


?7 


66 


• -T9“- 




U665* 


51 


70 


-19 




R147* 


49 


68 


-19 


To test an idea, try it 


U624* 


54 


73 


-19 


U629* 


50 


69 


-19 




U675* 


56 


75 


-19 




U633* 


46 


65 


-19 




R120* 


53 


72 


-19 


Transfer of momentum between objects 


U634* 


44 


63 


-19 


R109 


66 


86 


-20 


Rocks on Earth's surface are solid 


U618* 


57 


77 


-20 




R152 


33 


53 


-20 


Because of vaccinations few people get 
smallpox 


R150* 


35 


55 


-20 


Fran chart — sodium is least carmon in 
human body 


U667* 


43 


63 


-20 




U616* 


60 


80 


-20 




U661 


67 


87 


-20 




U611* 


67 


87 


-20 




U639* 


33 


54 


-21 




R123* 


44 


66 


-22 


What scientists learn from fossils 


R116* 


57 


79 


-22 


Mushrooms do not have green leaves 


U668* 


41 


63 


-22 



* Exercises also identified as atypical before balancing. 

: line separates atypical, high from atypical lew exercises. 
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Note 



The difficult exercises are often tricky and even misleading 
to respondents at the given age. For example, exercise R141 
is as follows: "A pint of water at a temperature of 50° 

Fahrenheit is mixed with a pint of water at 70° Fahrenheit. 

The temperature of the water just after mixing will be about: 

( ) 20° F.; ( ) 50° F. ? ( )60° F. ; ( )70° F. ; ( )120° F. ;' ( jl 
don't know." Nine-year-olds are apt to be confused by the 
references to Fahrenheit, pints and the ° notation. Most 
(69%) treated the problem as one requiring addition of the two 
numbers, and responded 120° F. Of the national sample, 12% 
responded "I don't know," and only 7% responded with 60° F., 
the correct answer. Such difficult exercises do not tend to 
differentiate well between different subgroups of respondents, 
in particular between Blacks and Non-Blacks. About all one 
can say is that on such exercises. Black and Non-Black 
respondents are about equally often misled. 

There are a couple of exercises where other factors may be at 
work behind the relatively good Black performance level. 
Exercises U621 and U617, both unreleased, require pieces of 
information about nutrition and volcanoes, respectively. The 
particular information involved is quite standard in first- 
and second-grade curricula, and the lack of color disparity 
is perhaps not too surprising. A third exercise, R142, 
requires the children to level a balance by hanging a weight 
appropriately on the lighter arm and reading off the proper 
numerical indicator on the arm. This was a trial-and-error 
exercise with actual weights rather than a paper-and-pencil 
question, and on such a directly experiential task. Black 
children did as well as white children. In fact, both groups 
exceeded 96% success. 

The result from this last exercise hints that Black performance 
at the youngest age level is improved for exercises with direct 
experiential content. Furthermore, the Black disadvantage is 
great for exercises far removed from daily experience. The 
lower section of Exhibit 3-3 lists the exercises with atypically 
low Black performance. Four of the unreleased exercises involve 
relatively remote textbook facts about astronomy, geology, 
meterology, chemistry, electricity, human biology and the 
behavior of animals. Two call for using symbols and graphs, 
and one refers to the activities of scientists. The released 
exercises in Exhibit 3-3 convey the specific flavor of the 
kinds of questions on which Black 9-year-olds perform most 
poorly relative to other 9-year-olds. Exercise R116, with a 
Black disadvantage of 22%/ reads, "Which of these plants does 
NOT have green leaves: ( )A dandelion; ( ) Grass; ( )A mushroom; 

( )A willow tree; ( )I don't know." Another exercise (R123), 
with a disadvantage of 22 % , is, "What can scientists learn by 
studying fossils : ( )Why earthquakes took place; ( )What 
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animals lived long ago; ( ) How far the moon is from Earth; 

( )What the weather will be tomorrow; ( )I don't know." This 
exercise, besides covering a content area quite removed from 
daily experience, requires that the children understand the 
vocabulary item "fossils." Other released exercises with a 
notable vocabulary component are R152, concerning vaccinations, 
on which the Black disadvantage is -20%, and R109, which 
refers to the Earth's "surface" (disadvantage also -20%). 

Two other released exercises listed in Exhibit 3-3, R147 and 
R120, pertain to rather disparate contents: how the respondent 

might best tell if salt water taffy is really made just by 
mixing water, salt and sugar (mix those things and see what 
happens) ; and what would happen to a wooden block hanging on a 
string if another such block were released from the left in an 
arc to hit it (it will swing off to the right) . These two 
exercises both require an abstract attitude toward unusual uses 
of familiar objects--a detached "What If" approach. If such an 
attitude is indeed needed to achieve correct responses on these 
two exercises, it apparently does not come as easily to Black 
9-year-old respondents. 



13-Year- Olds 



Exhibit 3-4 lists the exercises on which Blacks perform 
atypically at age 13. Of these fifteen exercises on which 
Blacks perform atypically well, five have national percent 
correct figures below 20%, and another three between 20 and 
40%. Thus there is some indication that difficult, undifferen- 
tiating exercises occur at the 13-year-old as well as the 9-year 
old level. One or two exercises in this group, however, deserve 
note as being different from the difficult Science achievement 
questions discussed for 9-year-olds. Released exercise R247, 
with a Black advantage of 3%, is, "Do you ask questions about 
why things in nature are the way they are: ( ) Often; 

( )Sometimes; ( )Never...If so, what questions have you asked 
most recently?" An "often" questioning attitude is reported by 
a slightly greater proportion of Black than Non-Black children. 
Unreleased exercise U775 is similar in its import. Whether this 
reported curiosity is specific to Science is not clear. 



Exhibit 3-4 



Exercises showing atypical effects, balanced, 
for Blacks for age 13 



% Correct 



Exer- 
cise # 


Black 


Nat'l 


% l in- 
ference 


Content 


R247* 


10 


7 


3 


Do you often question things in 


U774* 


20 


19 


1 


nature 


U747* 


8 


8 


0 




U741* 


34 


34 


0 




U775* 


5 


5 


0 




R222* 


38 


38 


0 


By natural selection, why giraffes 


U769* 


66 


66 


0 


have long necks 


U767* 


89 


89 


0 




R237* 


34 


35 


-1 


Time a pendulum's swings 


R243* 


67 


68 


-1 


Measurements in science: close not 


U729* 


50 


51 


-1 


identical 


U766 


9 


11 


-2 




R217* 


46 


48 


-2 


Burning gasoline in a car creates 


R202* 


96 


98 


-2 


heat 

Why brush your teeth 


R203* 


90 


92 


-2 


Thick, dark clouds bring rain 


U709 T 


61 


80 


-19 


U750* 


60 


79 


-19 




R241 


60 


79 


-19 


Mathematics a useful skill in 


R234* 


43 


62 


-19 


science 

Balance beam — weight in pan; weight 


U754* 


44 


63 


-19 


on hook 


R205* 


66 


85 


-19 


Comfortable temperature: 70° F 


R236* 


40 


60 


-20 


From chart — determine dog's food 


R212* 


38 


59 


-21 


ration 

Movement of air nesses predicts 


U707* 


63 


84 


-21 


weather 


R214* 


33 


54 


-21 


Counterbalancing unequal weights on beam 


R206* 


57 


78 


-21 


Why fan a campfire 


U717* 


41 


63 


-22 




U760* 


19 


42 


-23 




U758* 


26 


49 


-23 




U759* 


23 


47 


-24 




U727* 


27 


53 


-26 




U753* 


43 


69 


-26 






Exhibit 3-4 (Continued) 



% Correct 



Exer- 

cise# 


Black 


% Dif '• 

Nat'l ference 


R213* 


30 


57 


-27 


R233* 


43 


70 


-27 


R215* 


24 


52 


-28 



Content 

Radio waves least upset plant and 
animal life 

From chart — compare guinea pig 
weights 

Flower seeds develop from ovules 



♦Exercises also identified as atypical before balancing. 

Note: line separates atypical high from atypical low exercises. 



Several of the exercises on which age 13 Blacks do about as 
well as Non-Blacks have reference to direct experience. 

Exercise R217 is, "Most of the chemical energy of the gasoline 
burned in a car is not used to move the car but is changed into: 

( ) electricity ; ( )heat; ( ) light; ( )magnetism; ( ) sound; ( )I 
don't know." Answer: heat. Exercises R202 and R203 ask the 

purpose of toothbrushing, and what kind of clouds bring rain. 

Two unreleased exercises, U729 and U767, are questions based on 
everyday experiences about human biology and about scientists, 
respectively. In R237, the respondent is given a stopwatch and 
asked to time ten swings of a pendulum. 

When we turn to those exercises on which Black 13s perform 
especially poorly in the lower section of Exhibit 3-4, there is 
evidence of the same bookish and abstract qualities noted 
earlier for poor performance of Black 9-year-olds. Among 
released exercises, there is a vocabulary component for R215, 
where the correct answer is "ovules"; R205, where the most 
comfortable "Fahrenheit" temperature is solicited; and R212, 
where information was asked about predicting weather, with the 
correct answer, "The movement and characteristics of air masses." 
Black deficits on these exercises are 28%, 19% and 21%, 
respectively. Unreleased exercises U707, U717 and U727, 
with comparable deficits, involve terms that may also be 
relatively unfamiliar to Black children. 

Another set of exercises with poor Black performance involves 
the plotting and interpretation of data graphs. On Exercise R236, 
the child is shown a month-by-month chart of the weight of a dog. 
He is given recommended directions for feeding a dog according 
to its weight, and must then specify what food ration is appro- 
priate in a given month. On Exercise R233, a study of the daily 
weights of two guinea pigs is described, and the respondent must 
pick the verbal statement which corresponds accurately with a 
graphical summary of the results. Unreleased exercises U758 and 
U760 require the child to use physical apparatus to generate data 
which is to be graphed, and then to label and interpret the 
graph. On all four of these exercises, the deficit exceeds 20%. 
Another two exercises involve anticipation or calculation of the 
appropriate way to balance a beam. On Exercise R214, the child 
must interpret a sketch of a lever arm and apply the principle 
that a 2-lb. weight four units from the pivot will balance a 
4“lb. weight two units away on the other side of the pivot. On 
Exercise R234 , the respondent is given a balance pan and some 
weights, along with a pictorial representation of this apparatus, 
and is asked three questions about balance points and weights. 

On the first question, performance of Black 13-year-olds is 
atypically poor (19% below national) . 



The remaining exercises in the lower section of Exhibit 3-4 
represent an assortment of contents. Exercise R213 poses an 
ecological question in which the respondent must pick the one 
of five given actions which will not upset nature's balance, 
the correct choice being the broadcasting of radio waves. 
Exercise R206 inquires why fanning helps campfires, and R241, 
"Which of the following is most useful in scientific research: 

( )music; ( )magic; ( )marketing; ( ) mathematics ; ( )manu- 
facturing." The unreleased exercises not thus far discussed 
include U709, which deals with astronomy, U754, posing a 
hypothetical measurement problem, U750 and U753, exploring the 
difference between a "fact" and an "opinion" and U759, which 
involves a rough numerical estimation. 

In general Black 13s do most poorly relative to the national 
sample on exercises involving either unfamiliar terms or 
remote content materials for which the correct approach 
involves the detached indirection characteristic of "scientific 
method. " 



17-Year-Olds 



At the 17-year-old level, there is a reappearance of the same 
factors accounting for atypically good and atypically poor 
performances among Blacks at other age levels (Exhibit 3-5) . 

In the upper group of exercises, we find a few difficult, 
undifferentiating exercises (U835, R332, U843) . Unreleased 
exercise U841 is also quite difficult, but appe.ars to differen- 
tiate groups in that Black respondents perform about 17% better 
than national average, after balancing (and also before 
balancing) . Since this is by far the greatest Black advantage 
on any Science exercise in National Assessment, one would like 
to be able to point to some special content factor involved in 
this exercise. Unfortunately, it is likely that fortuitous 
features of the format are responsible for the unique charac- 
teristics of this exercise. Several graphical alternatives 
are offered as completions for a question on a physical 
equation. The exercise is of such difficulty that respondents 
probably chose among the graphs on the basis of their superficial 
appearances. 



Exhibit 3-5 



Exercises showing atypical effects, balanced, 
for Blacks for age 17 



% Correct 

Exer- % Dif- 

cise # Black Nat'l ference Content 



1^41* 


36 


19 


17 




rj<.: + 


CC 


48 


8 


Efficient use of food can cause overweight 


U835* 


33 


26 


7 




R332* 


24 


20 


4 


Hew scientists determine rock age 


U843* 


18 


15 


3 


U856* 


42 


60 


-18 




U855* 


43 


62 


-19 




U813* 


48 


67 


-19 




U811* 


52 


71 


-19 




R312* 


40 


59 


-19 


Unaided eye detects certain wavelengths 
of light 


U809* 


51 


72 


-21 




U816* 


37 


60 


-23 




R316* 


29 


53 


-24 


Adrenalin is a stimulant to the heart 


R309* 


41 


65 


-24 


Who proposed natural selection in evolution 


R305* 


52 


76 


-24 


Movement of air nesses predicts weather 


R344* 


30 


55 


-25 


Time a pendulum's swings 


R340* 


54 


79 


-25 


Which weight experiment gives strongest 
evidence 


R341* 


49 


74 


-25 


Balance beam — weight in pan; weight on hook 


U820* 


29 


56 


-27 




U852* 


40 


73 


-33 





♦Exercises also identified as atypical before balancing. 

Note: line separates atypical high from atypical lew exercises. 
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Exercise R322 is also a tricky one. The correct alternative 
is the one which completes the following seemingly paradoxical 
sentence: "If a person who is a light eater has a tendency to 

be overweight, it is most likely due to highly efficient 
utilization of food by the body." We do not know what fraction 
of the national 48% correct is due to lucky misunderstanding, 
such as reading "inefficient" for "efficient" and thinking this to 
lead to overweight. Therefore, any interpretation of the Black 
advantage of 8% is hazardous. 

Three exercises of interest yield very slight Black advantages 
after balancing, but do not exceed the upper cutoff and are 
therefore not shown in Exhibit 3-5. One of these is released 
exercise R350. It reads, "Whenever scientists carefully 
measure any quantity many times, they expect that: ( )all of 

the measurements will be exactly the same; ( ) only two of the 
measurements will be exactly the same; ( jail but one of the 
measurements will be exactly the same; ( jmost of the measure- 
ments will be close but not exactly the same; ( )I don't know." 

The balanced proportion of Black 17s choosing the correct and 
eminently sensible answer (most will be close but not exactly 
the same) was 2.7% above the national average. An unreleased 
question (U807) about physical principles in home appliances 
yields a balanced Black advantage of 1.6%, and a released 
exercise (R314) about why giraffes evolved long necks, a 
balanced Black advantage of 1.5%. The latter is the same 
exercise as R222 for 13-year-olds, on which the balanced Black 
disadvantage was 0%. 

In the lower group of exercises (Exhibit 3-5) , vocabulary 
factors again account for some of the results. Released 
exercise R316 requires the knowledge that "adrenalin stimu- 
lates the heart," and unreleased exercises U809 and U820 also 
use unfamiliar words involving aspects of body chemistry. 

Exercise R309 asks for the association of Darwin's name with 
the idea of natural selection in a theory of evolution. 

Exercise R305 is the same question as R212, about "movement 
and characteristics of air masses," on which Black 13s also 
showed atypically great disadvantage. 

Other exercises replicating poor 13-year-old results are: 

U856, which like U758 required a graph of data generated by 
the respondent from a physical apparatus; and R341, which like 
R234 gave the respondent a balance pan, weights and pictorial 
instructions and asked how to balance a weighted pan. A 
further question with the same apparatus is unreleased exer- 
cise U852, with a 17-year-old Black balanced disadvantage 
of 33%. Clearly, Black 17s experienced some difficulty with 
laboratory exercises (although the number of 17-year-old Black 
respondents exposed to each such exercise was small, about 80, 
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and therefore the percentage results are statistically 
unstable) . Another laboratory exercise giving an atypically 
large deficit is R344, involving the timing of ten swings of 
a pendulum. This particular exercise showed no Black disad- 
vantage in its age 13 form (R243) . Comparing the percentages 
of correct responses on the pendulum exercise for the two ages 
in Exhibits 3-4 and 3-5, it appears that there was no Black 
\ improvement from ages 13 to 17, but a sizable national improve- 

ment (about 20%) . It is not clear whether this result 
indicates that the progress of Black teen-agers suddenly stops 
on certain laboratory Science tasks, or whether there is some 
fortuitous peculiarity in the administration of this particular 
exercise. Exercise R340 presents hypothetical experiments 
involving the measured weights of two objects, with four 
observations on each weight in each experiment, and on this 
exercise, there is a Black disadvantage of 25%. 

The other five exercises in Exhibit 3-5 include R312, a 
released exercise comparing different forms of electromagnetic 
radiation, with the answer that only "certain wavelengths of 
light" can be detected with unaided human eyes. The unreleased 
exercises (U855, U811, U813, and U816) cover various topics 
in electricity, the mechanics of springs, properties of light, 
and the concept of a hypothesis. 



Adults 



At the Adult level, Exhibit 3-6, six exercises show atypically 
good Black performance; of these, two (R433 and U948) are so 
difficult as to be undifferentiating. Of the remaining four, 
the two released ones are as follows — R410: "An electric 

current in a copper wire involves mainly the movement of: 

( ) copper atoms; ( ) copper molecules; ( ) electrons; 

( ) neutrons; ( ) protons; ( )I don't know." — R418: "A 5-pound 

rock is dropped from a cliff 500 feet high. The longer the 
rock falls, the greater is its: ( ) acceleration; ( ) potential 
energy; ( ) speed; ( ) total energy; ( ) volume; ( )I don't know." 
The Black advantages, after balancing, on these two exercises 
are each 4%. Both have the character that the straight- 
forward answers are correct — "electrons" for the first, 
and "speed" for the second exercise. Black Adults tend slightly 
less often than Non-Black Adults to be tempted to choose the 
fancy incorrect alternatives. The remaining two exercises with 
atypically good Black performance are both unreleased, U968 and 
U921. Exercise U968 is an attitude question (Objective 4) 
emphasizing curiosity in everyday matters, and U921 is a 
question about astronomy verifiable by direct observation. 
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Exhibit 3-6 



Exercises showing atypical effects, balanced, 
for Blacks for Adults 



% Correct 



Exer- 
cise # 


Black 


Nat'l 


% Dif- 
ference 


Content 


U968* 


56 


45 


11 




R433* 


24 


15 


9 


How scientists determine rock age 


U948* 


14 


9 


5 


R410* 


66 


62 


4 


Current involves movement of electrons 


R418* 


55 


51 


4 


Speed of falling rock increases 


U921* 


67 


63 


4 




R4(55 t 


’ T4 


70 


-26 


Secondary result of vasectomy \ A 

Adrenalin is a stimulant to the heart ^ - 


R406* 


43 


69 


-26 


U919 


37 


64 


-27 




R403* 


57 


84 


-27 


Movement of air masses predicts weather 


U957* 


25 


54 


-29 


U956* 


23 


56 


-33 





♦Exercises also identified as atypical before balancing. 

Note: line separates atypical high from atypical low exercises. 
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Adult exercises with atypically poor Black performance include 
two released questions overlapping with 17-year-old exercises 
with atypically poor Black performance. R406 concerns 
adrenalin as a stimulant to the heart, and R403 the importance 
for weather prediction of the "movement and characteristics 
of air masses." Two unreleased Adult exercises also overlap 
with previous exercises and yield comparable results. On 
Exercise U956, which repeats U760 and U856 involving the 
graphing of laboratory data, there is a Black deficit of 33%. 
The follow-up graphical exercise U957 shows a deficit of 29%; 
the parallel question at age 13, U758, yielded a deficit of 
2 3%. Unreleased exercise U919 overlaps U813 concerning 
properties of light, and at both ages 17 and Adult there is 
atypically poor Black performance. The only newly appearing 
exercise in the lower portion of Exhibit 3-6 is R405, with 
Black Adults performing 26% less well than national, after 
balancing. This exercise concerns the consequences for Adult 
human males of tying off their main sperm ducts in a simple 
operation. The alternative answers were; " ( )The voice will 
eventually become high pitched; ( ) Fatty pads will gradually 
develop on the hips; ( ) Behavior will eventually become more 
effeminate; ( ) The hair will develop in longer strands than 
usual; ( )None of the above will occur; ( )I don't know." The 
question as formulated requires the answer "None of the above," 
and as such is perhaps more an assessment of a non-anxious 
attitude toward side effects of apparently sensitive medical 
procedures than a direct probe of the knowledge of the purpose 
of vasectomy. 



Summary 



In general, over all age levels of what is probably the most 
suggestive theme running through these atypical exercises, we 
may state the following; Blacks perform best on those science 
exercises most dependent upon daily experience and common 
knowledge and worst on those which involve a detached research 
attitude toward the objects and phenomena of Science . There 
are several factors in the Black educational, environmental 
and cultural situation that might explain such an outcome. It 
is not possible from a single cycle of National Assessment to 
choose among the many conceivable sets of explanatory factors. 
A number of research efforts over a period of years would be 
needed to begin to resolve these questions. If the science 
performance of Black youngsters improves steadily in the years 
ahead, however, the need to explain the deficit on the first 
National Assessment will recede in importance. We all hope 
that some of the many current innovations in methods and 



technology of teaching and in the organization of educational 
systems will produce substantial educational gains for all 
children, Black and Non-Black. Future National Assessment 
cycles will show the pattern and extent of any such improve- 
ments . 



\ 
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Chapter 4 

PARENTAL EDUCATION GROUPS 



Every respondent to NAEP exercises, at age levels 9, 13, 17 and 
Adult, was asked to indicate the highest year of schooling 
completed by his mother and his father. In this chapter, we 
discuss the percentages of correct responses to the science 
exercises for each of several levels of reported parental 
education: 

Level Definition 

0 Education of parents was not ascertained. 

1 Neither mother nor father was educated beyond 
8th grade. 

\ 

2 Either mother or father had some high school, 
but/ neither completed high school . 



3 Either mother or father completed high school, 

neither was trained beyond high school. 

4 Either mother or father was educated beyond 
high school. 

/ 

It should be stressed that the data available on education of 
parents (except as noted below for some 9-year-olds) is that 
reported by the respondent on the "tail-sheet" of the exercise 
booklet. If the respondents were in error in reporting the 
schooling of their mothers or fathers, they will be misclassi- 
fied on parental education. 



More often than at other ages, 9-year-olds failed to respond 
to the questions about education of their parents . In some 
cases where responses to these questions were omitted, 
exercise administrators provided the information from school 
records. In many other cases, the parents' education was not 
ascertained, and the respondents were classified at level 0 on 
parental education. 

The median number of respondents classified at every level of 
parental education for each age is shown in Exhibit 4-1. (For 
different booklets or "packages" of exercises, the actual 
number of respondents varies somewhat, above and below these 
median numbers.) It may be seen from Exhibit 4-1 that, for 
many 9-year-olds, the information was not ascertained. Also, 



Exhibit 4-1 



Median numbers of respondents per exercise 



for 


each 


level of 


parental education 








Level of 


parental education 




Age 


0 


1 


2 


3 


4 


9 


789 


125 


144 


567 


787 


13 


252 


109 


250 


762 


1049 


17 


39 


146 


347 


697 


959 


Adult 


51 


293 


159 


200 


189 
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it is likely that at age 9, more than at the other ages, 
respondents made errors in responding concerning the numbers 
of years of school for their mother or father. 

Before presenting any detailed results, it is of interest to 
consider whether, on typical Science exercises, performance 
differences are related to the reported level of parents ' 
education. For this purpose, Exhibit 4-2 shows graphically 
the median effect -- expressed as a difference from the median 
national percentage correct — for respondents at each level 
of parental education. At each age it is apparent that median 
performance is below the national median for levels 0 (Unascer- 
tained) , 1 (No High School) , and 2 (Some High school), rising 
above the national median for level 4 (some Education Beyond 
High School) . At level 3 (Graduated from High School) median 
performance is near the national median. On typical science 
exercises, respondents who report that at least one parent had 
substantial formal schooling perform better, as a group, than 
respondents who report less schooling for both parents. 

Throughout this chapter, we must remember that respondents 
classed at a given level of parental education may differ in 
many ways from those classed at other levels. In particular, 
the various levels of STOC (size and type of community — see 
Chapter 6) , sex, region and color may not be proportionately 
represented at each level of parents* education. Effects that 
are apparently related to parental education, then, in some 
cases may be more simply explained by differences in STOC or 
differences in color that are masquerading as differences in 
education. In Chapter 5 we shall return to this matter. For 
now, we simply present the data by level of parents' education 
without any adjustment for the possible effects of other 
variables. 



Level 1; Neither Parent Attended High School 



Exhibit 4-3 shows the distribution of effects at parents’ 
education level 1 for all science exercises at each age. At 
each age, the median percent correct is below that for the 
national sample. The median effect ranges from -7.2% at age 9 
to -11.8% at age 13. This tells us that, for the average 
science exercise, respondents whose parents failed to attend 
high school obtain correct answers from 7 to 12 percent less 
often than the national average for each age group. 



MEDIAN EFFECT IN PERCENT CORRECT 



n- — — ^ — — 1 — 

> 

I 

Exhibit 4-2 




MEDIAN EFFECTS FOR LEVELS OF PARENTAL EDUCATION AT FOUR AGES. 
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Exhibit 4-3 



Distributions of parental education effects, No High 
for all science exercises at four ages 




No. of 

Exercises 145 



122 



124 



119 



Median -7.2% 



-11.8% -8.4% 



-7.9% 



O 
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School 



10 



0 



-10 



-20 



-30 



The arrows in Exhibit 4-3 show the "cutoffs" beyond which 
exercises are considered atypical in their effect (for defini- 
tion of "cutoff" see Appendix C) . Some exercises are atypical 
in showing greater than usual effects of level 1, parents' 
education. Other exercises are atypical in showing lesser than 
usual effects, or effects in the direction opposite to the 
usual direction. For a few science exercises, respondents 
reporting that neither parent attended high school give correct 
responses at a rate higher than the national percent correct. 

The discussion of atypical exercises will be left to Chapter 5. 

Level 2: At Least One Parent Attended High School 

In Exhibit 4-4 appear the distributions of effects for level 2 
of parental education for all science exercises. At each age, 
median performance of respondents for whom neither parent 
completed high school is below the median performance for the 
nation as a whole, and the majority of all science exercises 
display the effect. The effect, however, is less marked for 
Adults than for the other three age groups. Exercises that 
show atypical effects fall in the extremes of the displays of 
Exhibit 4-4, beyond the cutoffs shown by arrows. 

Level 3: At Least One Parent Graduated from High School 

Exhibit 4-5 shows that, on most exercises, performance of 
respondents at parents' education level 3 is similar to that for 
the national sample. At age 13, there is a slight tendency 
for such respondents to perform less well than the average 
for all 13s, while at the Adult age, such respondents perform 
a bit better than the average for all Adults. Beyond the 
arrows in Exhibit 4-5 are only a few exercises for which effects 
of education level 3 is atypical. 

Level 4: At Least One Parent Educated Beyond High School 

Exhibit 4-6 displays the distribution of effects for 
respondents at parental education level 4. Median effects 
range from 5.1% at age 17 to 9.1% for Adults. At each age, 
the body of results attests to the higher performance on 
science exercises of those respondents with at least one 
parent with Post-High School Education. A relatively large 
number of science exercises show atypical level 4 effects. 

These are the exercises that fall beyond the arrows in 
Exhibit 4-6. Those that remain atypical for balanced 
results are presented and discussed in Chapter 5. 
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Exhibit 4-4 



Distributions of parental education, Some High School, 
for all science exercises at four ages 




No. of 

Exercises 145 



122 124 119 



Median -4.8% 



- 6 . 2 % 



-7.6% -1.6% 
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Exhibit 
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Level 0 : 



Parental Education Unascertained 



From Exhibit 4-7, it is clear that for respondents who fail to 
provide the requested information about their parents' 
education, performance on Science exercises tends to be below 
the national average. The median effect ranges from -4.6% at 
age 9 to -16.8% for Adults. The relatively small deficit at 
age 9 suggests that all levels of actual parental education 
may be represented in this group of children who are unable to 
say how far their parents continued in school. But the story 
is quite different for Adults. Indeed, for Adults at level 0, 
performance is lower than for Adults at level 1 (neither parent 
attended high school) . 



Parental Education Effects for Various Science Objectives and 
for Physical Science Versus Biological Science 



Median relative performance for the various levels of parental 
education are given separately for the four science objectives 
in Exhibit 4-8. It may be seen that the trend showing better 
performance with increasing levels of parental education 
appears to hold separately for each science objective at each 
of the four ages. In general, the trend is quite similar for 
the exercises designed to assess the various science objectives. 
There is some suggestion of smaller effects associated with 
parental education for Objective 4 (attitudes towards Science 
and scientists) , although the smaller numbers of exercises 
administered for this objective prompt us to be somewhat 
cautious about this finding. 

In Exhibit 4-9, educational effects are displayed separately 
for physical science exercises, biological science exercises 
and unclassified exercises. The trend of effects dependent 
upon parental education is essentially the same for all three 
classes of exercises. 
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Exhibit 4-7 



Distributions of parental education, Unascertained, 
for all science exercises at four ages 




I 

> 



Exhibit 4-8 

Median effects associated with parental education groups 
for science exercises classified by objective 



Level of parental education 



Age 


Objective* 0 


1 


2 


3 


4 


9 


1 


- 4.4 


-7.3 


-4.6 


0.7 


5.4 




2 


-6.2 


-6.7 


-3.5 


0.8 


6.9 




3 


-4.5 


-7.1 


-7.3 


0.1 


6.2 




4 


- 3.7 


-9.0 


-5.4 


-0.4 


4.9 


13 


1 


-7.1 


-11.5 


-6.2 


-1.5 


5.4 




2 


-10.4 


-12.5 


-7.5 


-1.4 


5.7 




3 


-11.0 


-13.5 


-3.3 


-1.3 


4.9 




4 


-4.4 


-2.0 


-2.6 


-0.2 


1.4 


17 


1 


- 7.7 


-8.0 


-7.1 


0.2 


4.9 




2 


-12.5 


-10.5 


-9.7 


0.0 


7.3 




3 


-4.5 


-6.3 


-3.9 


-0.8 


4.0 




4 


-5.0 


-1.7 


-6.3 


0.1 


4.6 


Adult 


1 


-15.6 


-8.2 


-1.2 


2.4 


9.2 




2 


-18.1 


-7.2 


-5.2 


5.9 


6.2 




3 


-17.1 


-10.6 


-5.6 


2.0 


11.2 




4 


-11.2 


-5.3 


-.4 


0.0 


5.4 


*1 - Facts and 


principles 










2 - Abilities 


and skills 











3 - Understand investigative nature 

4 - Attitudes and appreciations 



0 
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Exhibit 4-9 



Median effects associated with parental education 
groups for physical science, biological 
science, and unclassified exercises 



Level of parental education 



Age P, B, or U* _0 



9 P -4.5 

B -4.4 

U -4.7 

13 P -7.4 

B -7.8 

U -9.2 

17 P -7.2 

B -12.6 

U -5.0 

Adult P -15.6 

B -17.8 

U -16.8 



12 3 4 



-9.4 


-4.4 


0.5 


6.2 


-5.9 


-3.9 


0.8 


5.0 


-7.1 


-8.1 


-0.3 


7.0 


-11.1 


-6.4 


-1.4 


5.2 


-12 .2 


-6.2 


-1.9 


5.4 


-12.3 


-3.6 


-0.9 


3.9 


-9.4 


-8.4 


0.1 


5.3 


-6.5 


- 7.2 


0.7 


4.9 


-7.6 


-6.3 


-0.1 


4.6 


-7.6 


-2.5 


3.7 


7.6 


-8.7 


-0.8 


2.0 


10.2 


-7.0 


-2.8 


0.3 


11.0 



*P - Physical Science 
B - Biological Science 
U - Unclassified Exercises 
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Chapter 5 



PARENTAL EDUCATION GROUPS: BALANCED RESULTS 



The previous chapter presented results on science exercises 
for parental education groups as they are . Within the group 
characterized as level 1 on parental education (where neither 
parent attended high school) we have relatively heavy 
representation of respondents from the inner city and from 
rural communities, of Blacks, and of Southeasterners . When we 
consider balanced effects for level 1 respondents, we are 
asking how their performance would be expected to compare with 
national performance if the proportion of inner city and rural 
respondents, the proportion of Blacks, and proportion of 
Southeasterners were the same in level 1 of parental education 
as for the national sample. The balanced results give a 
clearer impression of differences associated with parental 
education after removing effects that might be attributed to 
the classification of respondents by region, type of community 
or color. 

Exhibit 5-1 provides some information about changes in the 
results for science exercises as a consequence of balancing. 

We consider the difference between the medians at the highest 
and lowest levels of parental education for each of the four 
age groups. After balancing, that difference is reduced by 
from 2 percentage points to almost 6 percentage points. After 
balancing, the difference between median performance at educa- 
tional level 4 and at educational level 1 is relatively uniform 
over ages, ranging between about 9% and 12%. The conclusion is 
the same if we look at balanced results only for those exercises 
common to two or more ages. While some exercises show greater 
or lesser effects of parental education, the difference between 
median balanced performance of respondents who report that a 
parent was educated beyond high school and respondents who 
report that neither parent attended high school is roughly 10% 
at all age levels. 

Let us now turn to a more detailed perusal of results for each 
level of parental education. In particular, we consider 
science exercises with effects differing markedly from the 
overall median effect. 
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Exhibit 5—1 



Percentage Differences between median effects at parental 
education levels 4 and 1 for unadjusted and 
balanced results at each age 



A2£ 



Unadjusted Balanced Change due to 

difference difference balancing 



9 

13 

17 



13.0% 


9 


17.0% 


11 


13.6% 


11 


17.0% 


12 



.1% 3.9% 

.2% 5.8% 

. 6 % 2 . 0 % 

.4% 



Adult 



4.6% 



Level 1: Neither Parent Attended High School 



For level 1, the balanced median deficit from the national 
percentage correct varies from about 5% (at age 9 and at 
Adult) to about 7.5% (at ages 13 and 17). However, at each 
age, some science exercises show atypical ly greater effects and 
some atypically lesser effects. Data about these atypical 
exercises appear in Exhibit 5-2. 

At age 9, two released exercises (R149 and R120) show level 1 
deficits of 19% and 22%, respectively. R149 requires 
scientific reasoning from a pictorial representation. At 
neither age 13 nor 17 are there released exercises that show 
an atypically large deficit at level 1. At the Adult age level, 
R407 and R411 show large deficits, between 14% and 15%, 
whereas the median deficit is about 5%. Both require the 
mastery of knowledge — i.e., that testes produce sperm, and 
that flower seeds are produced by ovules. 

These exercises, and those unreleased exercises that show 
atypically large level 1 deficits, seem to fall into two 
classes: either (a) scientific knowledge is required and 

relatively unfamiliar words must be recognized (e.g., ovules, 
testes, photosynthesis) or (b) a cognitive task of translation 
from graphical or pictorial evidence to a conclusion stated 
in words is required. Respondents at this lowest parental 
education seem to have special difficulty with these two kinds 
of science exercises. 

Certain other exercises (R319 and R418) are atypically easy 
for respondents at level 1 of parental education; indeed, 
these respondents obtain correct answers more often than the 
national average. A careful review of the content of these 
exercises does not lead us to any simple hypothesis concerning 
why they appear to be unusually easy for respondents whose 
parents failed to attend high school. 



Level 2: At Least One Parent Attended High School 



Level 2 respondents tend to perform less well than the national 
average. The balanced median effects show deficits of from 
1% (for Adults) to 5% (for 17-year-olds). 

Inspection of exercises atypical at level 2 (Exhibit 5-3) 
suggest that the same principle may be at work here as at 
level 1. Exercises requiring reasoning from graphs or pictures 
tend to be exceptionally difficult for respondents whose parents 
failed to complete high school, as do exercises that demand 
special vocabulary knowledge. 



Exhibit 5-2 



Exercises showing atypical effects, balanced, 
for education level 1 
(Neither parent attended high school) 



% Correct 





Exer- 


Level 




% Dif- 




Age 


cise # 


1 


Nat'l 


ference 


Content 


9 


U621* 


84 


76 


8 






U648* 


19 


37 


-18 






U665* 


52 


70 


-18 






R149* 


45 


64 


-19 


Fran chart — select highest temperature 




H120* 


50 


72 


-22 


Transfer of monentum between objects 


13 


U769* 


79 


66 


13 






R240* 


14 


4 


10 


Find density of wood block using ruler 












and balance 




U741* 


42 


34 


8 




17 


R319* 


65 


54 


11 


Speed of falling rock increases 




U843* 


25 


15 


10 






R330* 


38 


29 


9 


Egg release is 14 days after menstruation 




U820* 


33 


56 


-23 






U853* 


33 


65 


-32 




Adult 


R418* 


62 


51 


11 


What constitutes chemical change 




U960 


38 


29 


9 






U919* 


70 


64 


6 






U956 


59 


56 


3 






U917 7 


56 


71 


-14 






U929* 


35 


49 


-14 






R407* 


54 


68 


-14 


Testes produce sperm 




R411* 


48 


62 


-14 


Flcwer seeds develop from ovules 




U927* 


33 


50 


-17 





♦Exercises also identified as atypical before balancing. 

Note: line separates atypical high from atypical low exercises. 



Exhibit 5-3 



Exercises showing atypical effects, balanced, 
for education level 2 
(At least one parent attended but failed 
to complete high school) 



% Correct 



h31 

9 

13 



17 



Adult 



Exer- 
cise # 


Level 

2 


Nat'l 


% Dif- 
ference 


Content 


R137* 


46 


33 


12 


Mercury is heavier than water 


U724* 


45 


58 


-13 




U723* 


45 


58 


-13 




R215* 


39 


52 


-13 


Flower seeds develop from ovules 


R220* 


26 


39 


-13 


Wood floats higher in salt water than 


U742* 


15 


28 


-13 


fresh 


U721* 


46 


60 


-14 




U760* 


28 


42 


-14 




R233* 


55 


70 


-15 


Fran chart — ccnpare guinea pig weights 


U758* 


34 


49 


-15 




R327* 


37 


33 


4 


Molecules of air carry sound 


U839* 


24 


20 


4 




R351 


13 


27 


-14 


Kinetic-molecular theory 


U853* 


50 


65 


-15 




U828* 


23 


39 


-16 




R309* 


49 


65 


-16 


Who proposed natural selection in evolution 


U814* 


49 


65 


-16 




U958 


37 


50 


’ -13 ‘ 




R440* 


38 


51 


-13 


Effect of reduced rabbit population on 


U956* 


42 


56 


-14 


hawks and grass 


R418* 


34 


51 


-17 


Speed of falling rock increases 


U960* 


11 


29 


-18 




U961* 


4 


23 


-19 





♦Exercises also identified as atypical before balancing. 

Note: line separates atypical high from atypical lew exercises. 
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Level 3: At Least One Parent Completed High School 



For respondents at this level, overall performance is close to 
the national average, with median balanced effects ranging from 
about -1% at age 13 to +1% for Adults. Inspection of atypical 
exercises (Exhibit 5-4) shows that there are few exercises at 
any age that depart markedly from the median performance, and 
that those few exercises seem not to be closely related, one to 
the other, in content or form. 

Level 4: At Least One Parent Educated Beyond High School 



For respondents at level 4 , median balanced performance on 
science exercises is well above the nation at all four ages, 
about 4% above for ages 9, 13 and 17, and about 7% above for 
young Adults. Many exercises are atypical in showing greater 
or lesser balanced effects as level 4; these are listed in 
Exhibit 5-5. 

Level 4 respondents performed especially well on exercises that 
demand a deep understanding of the scientific method — exer- 
cises concerning testing hypotheses (R147 and several unreleased 
exercises) , reading charts and graphs (R150, R233 and unreleased 
exercises) -- and on a number of exercises that demand knowledge 
of particular facts and principles. Where respondents at level 1 
show a deficit, respondents at level 4 tend to perform unusually 
well. 

Level 0: Parental Education Unascertained 



The balanced median effects for parental education level 0 are 
-4% at age 9, -5% at age 13, -9% at age 17 and -13% for Adults. 
The deficit in performance for respondents who fail to report 
the educational level of their parents becomes systematically 
larger with increasing age. It should be noted from Exhibit 
4-1 that only small numbers of 17-year-olds and Adults fall in 
parental education level 0. 

Exhibit 5-6 lists the exercises that show atypical differences 
from the median effect at each age. A number of exercises 
that show atypical deficits for level 0 seem to entail the 
ability to draw scientific inferences (e.g., R153, R131, R147, 
R235 , R342, U853, U857) . Others call for specific knowledge 
about science (U633, U723, R309, R316) . These exercises have 
some resemblance to those that show atypical deficit at level 1. 
It seems likely that of the respondents who fail to report the 
level of their parents' education, many have parents with 
relatively limited numbers of years of schooling. 
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Exhibit 5-4 





Exercises 


showing atypical effects, balanced. 








for 


education level 3 




(At 


least 


one parent 


completed high school) 






% Correct 








Exer- 


Level 




% Dif- 




Age 


cise # 


3 


Nat'l 


ference Content 


9 


none 










13 


U761 7 


35 


41 


-6 






R219 


34 


40 


-6 


Why milk is pasteurized 




U726* 


50 


56 


-6 






R213* 


51 


57 


-6 


Radio waves least upset plant and animal 












life 


17 


U814* 


71 


65 


6 






U860 


24 


30 


-6 






U856* 


53 


60 


-7 




Mult 


U963* 


32 


20 


12 






R405* 


81 


70 


11 


Secondary result of vasectomy 




U930 


40 


48 


-8 





♦Exercises also identified as atypical before balancing. 

Note: line separates atypical high fran atypical low exercises. 
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Exhibit 5-5 



Exercises showing atypical effects, balanced, 
for education level 4 

(At least one parent educated beyond high school) 



% Correct 



13 



17 



Exer- 
cise # 


Level 

4 


Nat '1 


% Dif- 
ference 


Content 


U633* 


75 


65 


10 




R159* 


33 


23 


10 


Writing a poem is not part of science 


R147* 


77 


68 


9 


To test an idea, try it 


U626* 


80 


72 


8 




R150 


63 


55 


8 


From chart — sodium is least common in 


U634* 


71 


63 


8 


human body 


R152* 


61 


53 


8 


Because of vaccinations few people get 


U646 


46 


38 


8 


smallpox 


R141* 


7 


7 


0 


Mixing water of 50° and 70° yields water 


U673* 


10 


11 


-1 


of 600 


U653* 


20 


21 


-1 




U717* 


72 


63 


9 




R213* 


66 


57 


9 


Radio waves least upset plant and animal 


U754* 


72 


63 


9 


life 


U773* 


45 


37 


8 




R233* 


78 


70 


8 


From chart — compare guinea pig weights 


U725* 


66 


58 


8 




R244* 


64 


56 


8 


What is a scientific theory 


U772* 


92 


92 


0 


U853* 


77 


65 


12 




R342* 


75 


66 


9 


Effect of reduced rabbit population on 


U829* 


45 


36 


9 


hawks and grass 


R316* 


62 


53 


9 


Mrenalin is a stimulant to the heart 


P343 


65 


56 


9 


Differentiate fact from theory 


R309* 


74 


65 


9 


Who proposed natural selection in evoluti 


R319* 


50 


54 


-4 


Speed of falling rock increases 



'AH 
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Exhibit 5-5 (Continued) 



% Correct 





Exer- 


Level 




% Dif- 




Age 


cise # 


4 


Nat'l ference 


Content 


Mult 


U958* 


75 


50 


25 






R419* 


71 


49 


22 


Hot water molecules move faster 




U927* 


69 


50 


19 






R426* 


55 


36 


19 


Molecular speed determines different 












states of water 




U920* 


82 


64 


18 






R407* 


86 


68 


18 


Testes produce sperm 




R423* 


58 


41 


17 


Classifying plants and aniirals by 












structure 




U965* 


73 


56 


17 






R409* 


78 


62 


16 


Who proposed natural selection in evolution 




U963 


18 


20 


-2 






U919 


62 


64 


-2 






U921 


61 


63 


-2 





♦Exercises also identified as atypical before balancing. 

Note: line separates atypical high frcm atypical low exercises. 
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Exhibit 5-6 



Exercises showing atypical effects, balanced, 
for education level 0 
(parental education unascertained) 



% Correct 





Exer- 


Level 




% Dif- 




Age 


cise # 


0 


Nat'l ference 


Content 


9 


U637* 


58 


57 


1 






U653* 


22 


21 


1 






R141* 


8 


7 


1 


Mixing water of 50° and 70° yields water 












of 60° 




R153 


30 


38 


-8 


Full sink would not explain faulty 












faucet 




R131* 


43 


51 


-8 


Cactuses lose little water through 












leaves 




U633* 


57 


65 


-8 






R147* 


59 


68 


-9 


To test an idea, try it 


13 


R208* 


79 


74 


5 


Rocket launch possible on moon 




U723* 


42 


58 


-16 






R235* 


45 


62 


-17 


Which weight experiment gives strongest 












evidence 


17 


U828* 


58 


39 


19 






U860* 


47 


30 


17 






R325 


57 


41 


16 


Function of placenta in pregnant watan 




U805 


50 


82 


-32 






R309* 


31 


65 


-34 


Who proposed natural selection in evolution 




R316* 


28 


53 


-35 


Adrenalin is a stimulant to the heart 




R342* 


30 


66 


-36 


Effect of reduced rabbit population on 












hawks and grass 




U853* 


25 


65 


-40 






R344* 


15 


55 


-40 


Time a pendulum's swings 




U857* 


14 


55 


-41 




Mult 


R445* 


26 


11 


15 


Find density of wood block using ruler 












and balance 




U961* 


34 


23 


11 






U957* 


9 


54 


-45 






U956* 


8 


56 


-48 





* Exercises also identified as atypical before balancing. 

Note: line separates atypical high from atypical low exercises. 

< 0 ° 
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Chapter 6 



SIZE AND TYPE OF COMMUNITY 

In previous reports respondents were divided into four size of 
community (SOC) categories: Big City, Urban Fringe, Medium City 

and Smaller Places. In this report a more detailed classifi- 
cation* into seven categories is used. This classification of 
respondents is by school not by pupil and is based in part on 
information from the questionnaire filled out by the principals 
of participating schools. See Appendix B of Report 1. 

The first of these new categories is Extreme Inner City. A 
school was classified as an Extreme Inner City school if a high 
proportion of parents were either not regularly employed or were 
on welfare, and only a low proportion were professional or 
managerial. The majority, 85.6%, of the respondents from 
Extreme Inner City schools came from the former Big City SOC 
category. 

The second new category is Extreme Affluent Suburb. A school 
was placed in this category if a high proportion of parents 
were professional or managerial and only a few were factory or 
farm workers, or not regularly employed. More than half of the 
respondents from Extreme Affluent Suburb schools came from the 
Big City Fringe (53.0%) SOC category. Big City (29.3%) and 
Medium City (16.5%) making up the rest. 

The third new category, Extreme Rural, was defined by schools 
where a high proportion of parents were farm workers. The 
number of respondents in this category coming from the SOC 
category of 8maller Places was 78.1%. 

These three extreme categories were defined so that approxi- 
mately 10% of the total sample fell into each. Thus, 70% of 
the respondents whose schools did not fall into one of these 
extreme categories were grouped by their original size of 
community classification. Thus, a student in the Big City 
classification was classified as Inner City Fringe if he did 
not fall in one of these new extreme categories. Students in 
the Urban Fringe who did not fall into a new extreme category 
were classified as Suburban Fringe. Respondents from the 
Smaller Rlaces category were classified as Small Cities if they 
did not fall into one of the new extreme categories. Since the 
Medium City Size Community category contributed few, if any, 



* See Appendix D for explicit definitions of these new classi 
f ications . 
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schools to the three new extreme categories, it was left 
virtually unchanged. 

Exhibit 6-1 displays the median effect (difference between the 
percent correct for the subgroup and the national percent 
correct) for each of the four original categories and for the 
new categories by each of the four age groups in the sample. 

It is clear from these data that the criteria for picking the 
three extreme types of communities did indeed isolate groups 
of students whose performance was extreme. 

The concentration of the Big City deficit in the Extreme Inner 
City is particularly striking. This large deficit and to a 
lesser extent that associated with the Extreme Rural category 
clearly identifies those schools where American education 
faces its greatest challenge. Obviously many of the children 
in these schools are seriously deficient in their science 
achievement. 

The gap between the Extreme Affluent Suburb and both the Extreme 
Inner City and the Extreme Rural is largest for 9-year-olds 
(7.2% vs. -15.1% and -6.3%). It decreases for the 13- and 17- 
year-olds (6.3% vs. -13.7% and -6.1%) and (5.1% vs. -7.4% and 
-3.5%), and then rises sharply for the young Adults (10.9% vs. 
-10.2% and -4.7%). Since the young Adults were not attending 
school, a different technique was used to classify them into the 
extreme subcategories. This, together with the tendency for the 
better educated to escape from the Extreme Inner City and for 
the most successful to migrate to the Extreme Suburb, probably 
underlies the large gap for the young Adult sample. 

Compared with Adults, 17-year-olds show markedly smaller deficit, 
in the Extreme Inner City and a smaller advantage in the Extreme 
Affluent Suburb. However, this may be due to the fact that there 
were substantially more very difficult exercises given to the 17- 
year-old sample than the other age groups (See page 1 0 Chapter 1 
of Report 1) . Very difficult multiple choice exercises tend to 
give the guesser an advantage over the respondent who tries to 
work it out. This hypothesis is supported by the fact that when 
we analyze the 69 exercises with national percents of at least 
40% for 17-year-olds, the national effects are: for Extreme 

Inner City -10.4%, for Extreme Affluent Suburb +6.2% and for 
Extreme Rural -6.1%. It may be that the 17-year-old respondents 
were not as representative of their age group as were those 
respondents at the other ages. 

The median effects for 9-year-olds for physical science exercises, 
biological science exercises and the remaining unclassified 
exercises are shown in Exhibit 6-2. This exhibit also displays 
the median effects for the four science objectives described in 
Report 1. The median performance of each of the seven community 
types is remarkably uniform across the different sorts of 
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Exhibit 6-1 



Median effects by size and type of community 

for all ages 

Age 



soc 


STOC 


9 


13 


17 


Adult 


Big city 




- 4.8 


- 4.7 


- 2.6 


- 2.4 




Extreme inner city 


- 15.1 


- 13.7 


- 7.4 


- 10.2 




Inner city fringe 


- 2.6 


- 3.8 


0.3 


- 2.9 


Urban fringe 




3.0 


3.4 


2.4 


3.2 




Extreme affluent suburb 


7.2 


6.3 


5.1 


10.9 




Suburban fringe 


2.7 


2.9 


1.0 


0.8 


Medium city 




0.8 


1.1 


0.8 


0.4 




Medium city 


0.8 


1.9 


1.2 


0.4 


Smaller places 




- 1.2 


- 1.1 


- 2.1 


- 2.8 




Extreme rural 


- 6.3 


- 6.1 


- 3.5 


- 4.7 




Small cities 


0.9 


0.5 


- 1.4 


- 2.7 



Note: Figures show difference between the group percent correct and the 
national percent correct. 



63 
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Exhibit 6-2 



Median effects for 9-year-olds by 
type of question and objective* 



Unclas- Objective 

Physical Biological sified 





science 


science 


exer. 


1 


2 


3 


4 


Extreme 

Inner City 


-15.4 


-15.7 


-13.4 


-15.7 


-17.1 


-11.5 


-13.4 


Inner City 
Fringe 


-3.1 


-2.6 


-1.8 


-3.1 


-0.6 


-3.0 


0.8 


Extreme 

Affluent 

Suburb 


7.3 


7.0 


9.7 


7.2 


7.2 


8.5 


2.6 


Suburban 

Fringe 


3.5 


2.0 


1.7 


3.0 


2.4 


1.8 


1.1 


Medium 

Cities 


0.4 


1.4 


1.0 


0.9 


0.6 


2.3 


0.2 


Extreme 

Rural 


-6.3 


-5.6 


-8.2 


-6.0 


- 7.2 


-9.4 


-6.2 


Small 

Cities 


0.9 


1.0 


1.7 


0.9 


1.2 


0.5 


-0.1 



*1 - Facts and principles 

2 - Abilities and skills 

3 - Understand investigative nature 

4 - Attitudes and appreciations 
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exercises. In particular* their relative performance on physical 
science exercises almost exactly matches their performance on 
biological science questions. The slight variability observed 
in the rest of the table is quite likely due to the small number 
of exercises included in these groups. Exhibit 6-2. 

Exhibits 6-3, 6-4, and 6-5 show comparable data for 13-year-olds, 
17-year-olds and young Adults, respectively. The 13-year-olds, 
like the 9-year-olds , show virtually no change in pattern be- 
tween physical and biological science or between objective one, 
two and three. Objective 4 shows a much smaller range of vari- 
ation but this is based upon only a few exercises. 

In the Extreme Inner City, 17-year-olds have a greater deficit 
in physical science than in biological science (-8 . 5% compared 
with -5. 3$) while those in the Extreme Rural show just the 
opposite effects (-3 . 1% compared with -6.8$. Thus, the Extreme 
Inner City students do relatively better in biological science 
and the Extreme Rural ones better in physical science. 

Like the 9-and 13-year-olds, young Adults show little difference 
in their relative performance on physical and biological science 
exercises except that the Extreme Inner City respondents showed 
a slightly smaller deficit in physical science. 

Exhibits 6-6 through 6-9 show the distribution of effects for 
all science exercises across all four ages for the three 
extreme categories and the Medium City category. A comparison 
of the medians for all seven STOC categories was made earlier 
in this chapter, and the four distributions selected for Exhibits 
6-6 through 6-9 were most interesting and indicative of the 
others . 



Exhibit 6-3 



Median effects for 13-year-olds by 
type of question and objective* 



Objective 

Physical Biological Other 





science 


science 


science 


1 


2 


3 


4 


Extreme 

Inner City 


-13.4 


-14.9 


-ii.i 


-14.1 


-14.0 


-14.6 


-3.5 


Inner City 
Fringe 


-3.5 


-5.4 


-3.0 


-4.1 


-4.1 


-3.0 


-2.2 


Extreme 

Affluent 

Suburb 


6.1 


7.4 


4.2 


6.2 


7.8 


5.4 


2.6 


Suburban 

Fringe 


3.0 


3.2 


2.0 


2.7 


3.3 


2.3 


1.4 


Medium 

Cities 


1.6 


3.4 


.3 


1.7 


2.6 


0.7 


0.9 


Extreme 

Rural 


-6.5 


-6.5 


-4.1 


-6.8 


-6.1 


-1.7 


-2.4 


Small 

Cities 


0.7 


-0.1 


1.1 


0.8 


-0.6 


2.2 


-0.1 



1 - Facts and principles 

2 - Abilities and skills 

3 - Understand investigative nature 

4 - Attitudes and appreciations 
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Exhibit 6-4 



Median effects for 17-year-olds by 





type of 


question and objective* 










Physical Biological 
science science 


Other 

science 


1 


Objective 
2 3 


4 


Extreme 

Inner City 


-8.5 


-5.3 


-6.6 


-8.0 


-7.9 


-5.4 


-4.1 


Inner City 
Fringe 


-0.4 


2.0 


0.3 


0.2 


0.0 


1.1 


1.2 


Extreme 

Affluent 

Suburb 


5.2 


4.8 


5.1 


4.9 


6.1 


7.0 


4.4 


Suburban 

Fringe 


0.6 


1.3 


3.5 


0.9 


2.3 


-0.7 


3.3 


Medium 

Cities 


1.2 


1.3 


-0.2 


1.4 


1.3 


1.3 


-0.6 


Extreme 

Rural 


-3.1 


-6.8 


-2.2 


-3.3 


-4.4 


-9.2 


0.8 


Small 

Cities 


-0.8 


-2.3 


-2.4 


1.4 


-1.6 


-1.2 


-2.4 



*1 - Facts and principles 

2 - Abilities and skills 

3 - Understand investigative nature 

4 - Attitudes and appreciations 



S7 



o 

PRir 
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Exhibit 6-5 



I 

Median effects for young Adults for 
type of question and objective 



Ob j ective 

Physical' Biological Other 





science 


science 


science 


1 


2 


3 


4 


Extreme 
Inner City 


-10.0 


-13.1 


-8.2 


-10. 0 


-13 . 5 


-8.5 


-7.1 


Inner City 
Fringe 


-2.9 


-3.1 


-2.3 


-3.5 


-0.9 


-3.6 


-1.0 


Extreme 

Affluent 

Suburb 


9.5 


10.0 


14.1 


11.0 


6 . 6 


13.0 


9.9 


Suburban 

Fringe 


0.6 


1.4 


-2.0 


0.3 


4 . 6 


-0.6 


-1.6 


Medium 

Cities 


1.2 


-1.0 


0.2 


-0.1 


2 . 2 


-0.2 


0.9 


Extreme 

Rural 


-4.0 


-5.1 


-7.3 


-9.9 


-8 . 0 


-7.6 


-8.9 


Small 

Cities 


-2.9 


-2.2 


-4.3 


-2.9 


-1.8 


-7.1 


-2.4 



1 - Facts and principles 

2 - Abilities and skills 

3 - Understand investigative nature 

4 - Attitudes and appreciations 
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Exhibit 6-6 



Distributions of Extreme Inner City effects 
for all science exercises at four ages 




Median -15.1% 



-13.7% - 7 . 3% 



- 10 . 2 % 



O 

ERIC 
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Exhibit 6- 7 



Distributions of Extreme Affluent Suburb effects 
for all science exercises at four ages 



Age 




II 

No. of 

Exercises 145 122 124 

Median 7.2% 6.3% 5.1% 





■ 




40 . 



30 



20 . 



10 . 



0 . 



- 10 . 
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correct less national % correct 



Exhibit 6-8 



Distributions of Medium City effects 
for all science exercises at four ages 




Median 0.8% 



1. 9% 



1. 3% 



0.4% 



Exhibit 6-9 



Distributions of Extreme Rural effects 
for all science exercises at four ages 
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Age 






I 



No. of 

Exercises 145 



122 



124 



Median 



-6.3% 



- 6 . 1 % 



-3.5% 
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Chapter 7 



SIZE -AND -TYPE OF COMMUNITY GROUPS: BALANCED RESULTS 



The median effects associated with the seven types of community 
after balancing on region, sex, color and parental education 
are presented in Exhibit 7-1. In general the balanced effects 
are just about half of the unadjusted effects presented in 
Exhibit 6-1 of the preceding chapter. However, there are 
several exceptions worth noting: the median balanced effects 

for 13-year-olds, 17-year-olds and Adults in the Inner City 
are only about one-third as large as the unadjusted medians, 
and that for Extreme Rural Adults is essentially unchanged 
after balancing. 

The large reduction in the type of community effects produced 
by balancing indicates that much of the unadjusted effect was 
due to masquerading by other variables. The especially large 
reductions for inner city respondents suggest that the problems 
facing the inner city schools come from the combination of 
several different sorts of factors, some associated with color, 
some with parental education and some with region. The effects 
remaining after balancing are primarily associated with features 
of the inner city community. Because a variety of factors con- 
tribute to the inner city deficit, there may well be a variety 
of ways to improve the achievement of the children in these 
schools . 

In a similar way, a number of factors acting in combination are 
associated with the performance deficit found for Extreme 
Rural communities and the performance advantage in the Extreme 
Affluent Suburb. 

Exhibit 7-1 clearly shows the inner city and rural deficit and 
the affluent suburb advantage for typical science exercises. 
However, not all science exercises show the same effects, and 
it may be instructive to consider atypical exercises, for which 
the size-type of community balanced effects differ from the 
median balanced effect for that community. 

Extreme Inner City 

Exhibit 7-2 lists atypical exercises for the Extreme Inner City. 
At each age, those exercises for which inner city performance is 
atypically high are difficult exercises where the national 
percentage correct is no greater than 34% (with only one ex- 
ception, apparatus exercise R341 at age 17) . The unusually good 
performance on these difficult exercises could be due to more 
frequent guessing on the part of inner city respondents. 



I" 

c 



•o 
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Exhibit 7-1 



Balanced median effects by size and 
type of community for each age 



Age 



Extreme Inner City 


9 

-6.9 


13 

-5.5 


±1 

-2.3 


Adult 

-3.2 


Inner City Fringe 


-1.5 


-2.2 


0.4 


-1.9 


Extreme Affluent Suburb 


4.4 


3.3 


2.6 


5.6 


Suburban Fringe 


0.7 


1.0 


-0.6 


-0.9 


Medium Cities 


0.4 


0.9 


0.6 


0.8 


Extreme Rural 


-3.4 


-2.8 


-1.6 


-4.5 


Small Cities 


0.5 


0.7 


-0.2 


-2.0 



r 
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Exhibit 7-2 



Exercises showing atypical effects, balanced, 
for Extreme Inner City for all ages 



% Correct 





Exer- 






% Dif- 




h31 


cise # 


EIC+ 


Nat'l 


ference 


Content 


9 


U655* 


23 


14 


9 






U673* 


17 


11 


6 






R140* 


19 


14 


5 


Dead plants form coal 




R137* 


37 


33 


4 


Mercury is heavier than water 




R149* 


46 


64 


-18 


Fran chart — select highest temperature 




R104* 


73 


91 


-18 


Iron doesn ' t readily bum 




U644* 


23 


41 


-18 






R125* 


42 


61 


-19 


Plants get water from soil 




U616* 


61 


80 


-19 






R146* 


55 


75 


-20 


Air leak produces flat tire 




U622* 


55 


75 


-20 






U665* 


49 


70 


-21 






U682* 


50 


72 


-22 






U630* 


45 


68 


-23 




13 


U741* 


40 


34 


6 






U758* 


32 


49 


-17 






U760* 


25 


42 


-17 






R209* 


46 


65 


-19 


Sedimentary rock in layers 


17 


R341 


85 


74 


11 


Balance beam — weight in pan; weight on 












hook 




U816 t 


45 


60 


-15 






R314 


42 


56 


-16 


By natural selection, why giraffes have 












long necks 




R306* 


45 


66 


-23 


What doesn't increase supply of food 


Mult 


U961 


1 


21 ' 


‘ -72 






U960 


6 


29 


-23 






U956* 


32 


56 


-24 






U957* 


26 


54 


-28 






R441* 


9 


48 


-39 


Time a pendulum's swings 




U955* 


28 


71 


-43 





♦Exercises also identified as atypical before balancing. 

’’‘Extreme Inner City 

Note: line separates atypical high fran atypical lew exercises. 





63 



Many more exercises show atypically low performance for inner 
city respondents, especially at age 9 and for young Adults. 
Included in this group at age 9 are a number of exercises that 
assess rather earthy aspects of Science (R104 , R125, R146, U644, 
U616 , U622, U665 and U630); correct answers to many of these 
come not from the classroom, but from everyday experiences of 
children — experiences, however, that are less likely to be en- 
countered by 9-year-olds in an inner city environment. At the 
adult ages, all exercises displaying atypical deficit are 
apparatus exercises. Perhaps, when apparatus exercises are 
presented to adults during an interview in their home, inner 
city adults tend to be somewhat less cooperative, and thus 
appear to perform poorly on such exercises. 

Ext reme Affluent Suburb 



In the affluent suburb, more exercises show atypically high than 
atypically low performance (Exhibit 7-3) . Both released and un- 
released exercises that display unusually large affluent suburb 
advantages tend to assess relatively abstract facts and prin- 
ciples of Science; the affluent suburbs seem to perform best on 
exercises that depend upon science knowledge learned in the 
classroom. 

Extreme Rural 



While relatively few exercises are atypical for rural respon- 
dents (Exhibit 7-4) , there is a tendency for those on which 
performance is atypically high to tap knowledge that is likely 
to be learned from general experience in rural communities. 

For example, the unreleased exercises on this list at the Adult 
level ask about (a) the importance of green plants, (b) the 
effects of heating water in containers of varying shapes and 
(c) the behavior of a wooden block floating on water. In 
contrast, an atypical rural deficit is seen for several exer- 
cises demanding graph reading, and for exercises R406 and R409, 
requiring considerable mastery of vocabulary as well as de- 
tailed formal knowledge. 

Other STOC Groups 

For the Inner City Fringe (Exhibit 7-5) , the Suburban Fringe 
(Exhibit 7-6) , the Medium City (Exhibit 7-7) and the Small 
City (Exhibit 7-8), relatively few exercises display atypical 
effects. No patterns of exercise content are easily discern- 
ible. 
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Exhibit 7-3 



Exercises showing atypical effects, balanced, 
for Extreme Affluent Suburb for all ages 



% Correct 





Exer- 






% Dif- 




Age 


cise # 


eas t 


Nat'l 


ference 


Content 


9 


U645* 


56 


40 


16 






R123 


81 


66 


15 


What scientists learn from fossils 




R153* 


52 


38 


14 


Full sink would not explain faulty 












faucet 




R128* 


68 


55 


13 


Sun is a star 




U639* 


67 


54 


13 






R130* 


66 


53 


13 


Comfortable temperature : 70°F 




U648* 


50 


37 


13 






U651* 


29 


34 


-5 






R140* 


9 


14 


-5 


Dead plants form coal 


13 


U760* 


69 


42 


27 






U758* 


68 


49 


19 






U755* 


51 


59 


-8 






R234* 


54 


62 


-8 


Balance beam-weight in pan; weight on 












hook 


17 


U866* 


50 


30 


20 






R326* 


51 


35 


16 


Salt in water lowers freezing point 




U823* 


61 


46 


15 






U838* 


37 


22 


15 






U822* 


68 


54 


14 






R318* 


65 


51 


14 


What constitutes chemical change 




U813* 


81 


67 


14 






R343 


45 


56 ‘ 


-El 


"1 bIffer^tTate - fact IratTi theory ~ 


Adult 


U962* 


63 


22 


41 






U956* 


90 


56 


34 






R441 


77 


48 


29 


Time a pendulum's swings 




U957* 


81 


54 


27 





♦Exercise also identified as atypical before balancing. 

"''Extreme Affluent Suburb 

Note: line separates atypical high from atypical low exercises. 
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Exhibit 7-4 



Exercises showing atypical effects, balanced, 
for Extreme Rural for all ages 



% Correct 

Exer- % Dif- 

Age cise # ERf Nat'l ference Content 



9 


R157* 


51 


39 


12 


Recognize an untestable statement 




U666 


76 


66 


10 




U675 t 


'57 


75 


-T8 ' 






U662* 


62 


81 


-19 




13 


U762* 


31 


22 


9 




17 


U853* 


49 


65 


-16 






U857* 


39 


55 


-16 




Adult 


U922* 


76 


62 


14 






R418 


62 


51 


11 


Speed of falling rock increases 




U928* 


59 


49 


10 






U963 


30 


20 


10 


Adrenalin is a itlmulant to the heart 




R406* 


40 


69 


-19 




R409* 


34 


62 


-28 


Who proposed natural selection in evolution 



♦Exercise also identified as atypical before balancing. 
"^Extreme Rural 

Note: line separates atypical high from atypical low. 




78 



66 



Exhibit 7-5 



Exercises showing atypical effects, balanced, 
for Inner City Fringe for all ages 



ERJC 

j>yS3S333 



% Correct 





Exer- 






% Dif- 




Age 


cise # 


ICF t 


Nat'l 


ference 


Content 


9 


R157 


28 


39 


-11 


Recognize an un testable statement 




U615 


68 


80 


-12 






U635 


49 


61 


-12 




13 


R234* 


77 


62 


• 15 


Balance beam— weight in pan; weight on 












hook 




U755* 


73 


59 


14 






U714 


79 


71 


8 






U751* 


60 


73 


-13 






R215* 


38 


52 


-14 


Flower seeds develop from ovules 


17 


R343* 


68 


56 


12 


Differentiate fact from theory 




U865* 


30 


20 


10 




Adult 


U962* 


47 


22 


25 






U964 


33 


18 


15 






R449* 


”19 


"28 ‘ 


-19" 


"oftenlratST' r7v7 icTenoe prograins ’ " 




U915* 


53 


74 


-21 





*Exercises also identified as atypical before balancing, 
t Inner City Fringe 

Note : line separates atypical high f ran atypical low exercises . 



i O 



67 



Exhibit 7-6 



Exercises showing atypical effects, balanced, 
for Suburban Fringe for all ages 







% Correct 








Exer- 


4 * 




% Dif- 




Age 


cise # 


sf t 


Nat'l 


ference 


Content 


9 


R123* 


74 


66 


8 


_What sci^entists^lea^_frqn fossils 




R150*" 


46 


55" 


" -9 ” 


Fran chart — sodTum is least common in 












human body 


13 


U755 


53 


59 


-6 






R237* 


29 


35 


-6 


Time a pendulum's swings 




U762* 


14 


22 


-8 




17 


R344* 


41 


55 


-14 


Time a pendulum's swings 


Adult 


R441* 


62 


48 


14 


Time a pendulum's swings 




U921* 


48 


63 


-15 





♦Exercise also identified as atypical before balancing. 
"^Suburban Fringe 

Note: line separates atypical high from atypical low. 



O 

ERIC 



CO 



68 



Exhibit 7 — 7 



Exercises showing atypical effects, balanced, 



for Medium City for all ages 



% Correct 





Exer- 






% Dif- 




Age 


cise # 


MC + 


Nat'l ference 


Content 


9 


R122* 


78 


71 


7 


Honey bees are helpful to man 




U640* 


£0_ 


53 


7 




13 


U744 


35 


28 


7 






U737* 


36 


42 


-6 




17 


R322* 


40 


48 


-8 


Efficient use of food can cause over- 












weight 




R327 


25 


33 


-8 


Molecules of air carry sound 




U824 


36 


45 


-9 




Adult 


U960 


47 


29 


18 






R441* 


64 


48 


16 


Time a pendulum's swings 




U957 


67 


54 


13 






U956 


68 


56 


12 





*Exercise also identified as atypical before balancing. 
^Medium City 

Note: line separates atypical high from atypical low. 





Exhibit 7-8 



Exercises showing atypical effects, balanced, 
for Small City for all ages 



Age 



% Correct 

Exer- % Dif- 

cise # SC^ Nat'l ference 



Content 



9 U635 68 61 7 



13 U754* 56 63 -7 

17 R345* 29 39 -10 Understand conservation - of natter' 

Adult U957 38 5? ~ ~-16 

U963 4 20 -16 



*Exercise also identified as atypical before balancing. 
+ Small City 

*, 

Note: line separates atypical high fran atypical low. 



O 
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Chapter 8 



REGIONAL GROUPS: BALANCED RESULTS 



In Report 4, the patterns of correct response to the science 
exercises were examined for consistencies and discrepancies 
among the nation's regions. In this chapter, wo examine the 
patterns in the regional samples after balancing for the dis- 
proportionate representation of four other characteristics-- S ex, 

size and type of community, education of parents, and color. 

To what extent can the regional effects noted in Report 4 be 
considered to grow out of this lack of proportionality? What 
effects remain after appropriate balancing procedures (described 
in Chapter 1 and in Appendix B) have been applied to the re- 
gional data? 

The median effects at the four ages for the four regions, both 
the unadjusted and the balanced, are pictured in Exhibit 8-1. 

Thus, the effects of balancing have reduced the range of the re- 
gional effects by about one-third. In general, the relative 
order of the medians within ages remains the same after balancing. 
The exceptions are the reversals of the order of the Northeast 
and the Central regions at ages 9, 13, and Adult; however, in 
none of these comparisons were the differences, either before 
or after balancing, large enough to be statistically reliable. 

The most striking difference between the unadjusted and the 
balanced effects is the sharp reduction in the Southeast deficit. 
Apparently, a substantial part of the deficit observed in the 
unadjusted data grows out of the fact that the effects of other 
characteristics are represented in different proportions in the 
different regional samples. On the other hand, a significant 
Southeast deficit (minimal for age 13) still remains after 
balancing. Also remaining is the relatively high median effect 
for adults in the West. 4 ' * 

The contrast between unadjusted and balanced effects presented 
in Exhibit 8-1 involves all the science exercises . •' In Report 4, 
the unadjusted effects were also examined in terms of groups of 
exercises classified according to the four science objectives 
and according to content, i.e., physical and biological science. 
The only instance in which the median effects for subgroups of 
exercises displayed a different picture from that pictured in the 
medians for vail exercises was in the comparison between Objective 
1 and Objective 2 exercises presented in Exhibit 3-5 of Report 4. 
In 11 of the 16 comparisons, the median for Objective 2 exercises 
was more extreme than the median for Objective 1 exercises. This 
! contrast remains after balancing, as illustrated in Exhibit 8-2. 

/ However, the absolute differences tend to be smaller, as would be 

expected as a result of the general reduction in the size of the 
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O 



MEDIAN EFFECTS IN % CORRECT 



EXHIBIT 8-1 



MEDIAN EFFECTS FOR THE FOUR REGIONS 
AT FOUR AGES BEFORE AND AFTER BALANCING. 



BEFORE BALANCING 



AFTER BALANCING 



AGE 

13 17 



ADULT 



AGE 

13 17 



ADULT 



3 

ERIC 




Exhibit 8-2 



Median effects, after balancing, for all exercises, 
Objective 1 exercises only, and Objective 2 exercises only. 
(Median effects, before balancing, for 
all exercises, in parentheses) 



All 



Grot®) 


Age 


exercises 


West 


Adult 


1.9 


(2.4) 


Northeast 


17 


1.5 


(2.6) 


Central 


13 


1.2 


(1.9) 


Central 


9 


1.0 


(1.6) 


Northeast 


9 


0.8 


(2.3) 


Northeast 


13 


0.8 


(2.0) 


West 


17 


0.7 


(1.7) 


West 


9 


0.6 


(0.0) 


Northeast 


Mult 


0.3 


(0.6) 


Central 


Mult 


0.3 


(0.7) 


Central 


17 


0.1 


(0.0) 


West 


13 


-0.8 


(-0.5) 


Southeast 


13 


-1.8 


(-4.7) 


Southeast 


Mult 


-2.4 


(-4.9) 


Southeast 


17 


-2.5 


(-4.9) 


Southeast 


9 


-3.1 


(-5.1) 



Medians for 



Obj. 1* Obj. 2* * Ccnparisont 



1.4 


2.0 


0.6 


bigger 




1.2 


1.9 


0.7 


bigger 




1.2 


1.3 


0.1 


bigger 




0.7 


1.5 


0.8 


bigger 




0.5 


1.9 


1.4 


bigger 




0.4 


2.1 


1.7 


bigger 




-0.1 


1.3 


1.4 


across 


zero‘d 


1.0 


-0.6 


1.6 


across 


zero^ 


0.6 


-0.5 


1.1 


across 


zero"^ 


0.1 


1.6 


1.7 


bigger 




0.2 


-0.1 


0.3 


across 


zero^ 


-0.7 


-0.8 


0.1 


bigger 




-1.2 


-4.2 


3.0 


bigger 




-2.2 


-2.5 


0.3 


bigger 




-2.2 


-3.1 


0.9 


bigger 


it 


-3.3 


-3.2 


0.1 


smaller 


*TT 



♦Objective 1 - Facts and principles 
**Objective 2 - Abilities and skills 

^Objective 2 compared to Objective 1 
^Exceptions to the rule 



o 
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effects. (The median of the absolute differences is 1.35% for 
the unadjusted median effects and 0.85% for the balanced median 
effects . ) 

The median effects presented in Exhibits 8-1 and 8-2 provide one 
way of viewing the results of balancing. As was true of our 
study of the unadjusted data, it is also instructive to examine 
not only the medians but also the distributions from which the 
medians are derived, to identify exercises falling so far toward 
the extremes of the distributions as to be considered atypical, 
and to ask whether or not it is possible to discover any systematic 
distinctions between exercises atypical above and those atypically 
below the median effects. 

In the sections which follow, we report the results of these 
analyses, taking the regions in order Northeast, Southeast, Central 
and West and then comparing the balanced data reported here with 
the unadjusted data in Report 4. 



Northeast 



The distributions of balanced Northeast regional effects are pre- 
sented in Exhibit 8-3. The exercises falling beyond the boun- 
daries marked by the arrows on Exhibit 8-3 and so defined as 
atypical, are identified in Exhibit 8-4, together with percent 
correct for national, adjusted percent correct for the Northeast, 
and the percent difference. 

In general, the picture presented by the balanced data for the 
Northeast is similar to that provided by the unadjusted data of 
Report 4. At ages 9, 13, and 17 the medians are about 1% above 
the national medians (0.8%, 1.5%) in contrast to median effects 
of about 2% for the unadjusted medians. These are small but 
statistically reliable effects. The median for Adults in the 
Northeast differ from Adults in the country as a whole. The 
same conclusion was reached on the basis of the median of the un- 
adjusted effects (0.7%). 

Of the 18 exercises identified as atypical, all but four were 
also identified as atypical in the distributions of unadjusted 
effects, and two of these were near the line marking atypical 
items in the unadjusted distributions. Six of the eight exercises 
classified under Objective 2 show an advantage for the Northeast. 
In contrast, only four of the nine exercises classified under 
Objective 1 show an advantage. In general, then, the summary 
statement of Report 4 applies also to atypical exercises after 
adjustment: "there appears to be a tendency for the Northeast 

to show an atypical advantage on a higher proportion of Objective 
2 than of Objective 1 exercises." 
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Exhibit 8-3 
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Distributions of Northeast regional effects, balanced, 
tor all science exercises at four ages 



Age 



■8 ir 







o__ 


0 






O 

P 


P 






0 

0 


H . 

fd 

G 




<*> 


O 




-M 


*H 

4-> " 


lol 


(/) 


fd 




rd 


G 




O 1 


x: 






4J 


o 






.H 




O 


O 




£ 


£ 




No. 


of 






Exercises 145 
Median 0.8% 



122 

0 . 8 % 



124 
1. 5% 



119 
0. 3% 



Exhibit 8-4 



Exercises showing atypical effects, balanced, 
for the Northeast for all ages 



% Correct 





Exer- 


North- 




% Dif- 




Age 


cise # 


east 


Nat'l ference 


Content 


9 


R128* 


64 


55 


9 


Sun is a star 




R153* 


46 


38 


8 


Full sink would not explain faulty 












faucet 




U669* 


59 


52 


7 






U667* 


70 


63 


7 






U626* 


79 


72 


7 






U646 


45 


38 


7 






U676 


47 


41 


6 






R134 


41 


46 


-5 


Houseflies spread serious diseases 




R135* 


33 


38 


-5 


Effect of submerged rock on water level 




U635* 


52 


61 


-9 




13 


U760* 


54 


42 


12 






U758* 


61 


49 


12 






R2l4 


“78“ 


"“37 


-5 


Matter EeEaves as iF rrade oF atoms 




R222* 


30 


38 


-8 


By natural selection, why giraffes have 












long necks 


17 


U861* 


38 


29 


' 9 






R339* 


7o 


86 


-6 


Reason why car engine stops 


Adult 


R420* 


54 


44 


10 


_Function of jolacenta injpregnant woman 




R441* 


31 


48 


-17 


Time a pendulum's swings 



*Exercise also identified as atypical before balancing. 

Note: line separates atypical high from atypical lew exercises. 
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Southeast 



Distributions of balanced effects for the Southeast region are 
presented in Exhibit 8-5. Exercises showing atypical balanced 
effects for the Southeast are listed in Exhibit 8-6. As in the 
case of the Northeast samples, the balanced data for the South- 
east tend to show a similar, if somewhat less extreme picture 
than the unadjusted data found in Report 4. Whereas the un- 
adjusted median deficits were about 5%, the balanced median 
deficits are about 2.5%. The median balanced effects, there- 
fore, are only half as large as the unadjusted median effects, 
but they are still large enough to indicate that performance 
in the Southeast tends to be below the national performance on 
science exercises. 

Altogether, there are 26 Science exercises identified as 
atypical after balancing, a number somewhat smaller than the 42 
reported before balancing. Of the 26 exercises, 17 were identi- 
fied as atypical before balancing and 9 were not. As in the 
case of the data for the Northeast, most of the fluctuations 
can be attributed to the influence of random sampling. There 
is, however, one difference between the lists of atypical 
exercises in the two reports worth noting. In report 4, we 
noted a difference in the relative frequency of exercises 
classified under Objective 2 exhibiting atypical advantage or 
disadvantage in the Southeast. 

All nine of the Objective 2 exercises identified as atypical 
showed a disadvantage. in contrast, of the six Objective 2 
exercises showing atypical effects for the Southeast after 
balancing, three show an advantage and three a deficit. Thus, 
the general tendency for Objective 2 exercises to show a 
greater disadvantage than Objective 1 exercises in the South- 
east, as reported in Exhibit 8-2, cannot be attributed to the 
influence of a few atypical exercises; the effect must be a 
general one involving the Objective 2 exercises as a group. 

Central 



Distributions of balanced effects for the Central region are 
shown in Exhibit 8-7, and those exercises identified as atypical 
after balancing are listed in Exhibit 8-8. The medians at ages 
9 and 13 are less extreme than before balancing (1.0% and 1.2% 
in contrast to 1.6% and 1.9%, but it is still possible to say, 
on the basis of the balanced data, that the Central region 
performs about 1% better than the nation as a whole at ages 9 
and 13 . At ages 17 and Adult, the changes as a result of 
balancing are small, a reasonable result in view of the fact 
that the effects before balancing were small. 
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Exhibit 
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Exhibit 8-6 



Exercises showing atypical effects, balanced, 
for the Southeast for all ages 



% Correct 





Exer- 


South- 




% Dif- 




Age 


cise # 


east 


Nat'l ference 


Content 


9 


R140* 


24 


14 


10 


Dead plants form coal 




U677* 


40 


33 


7 




U653* 


26 


21 


5 






U638* 


61 


56 


5 






R138* 


39 


35 


4 


Clearing skies follow cold front 




R154* 


20 


16 


4 


Water freezing point same as ice melting 












point 




R152 


57 


53 


4 


Because of vaccinations few people get 
smallpox 




U635 


51 


61 


-10 




R156* 


69 


79 


-10 


Recognize a scientific explanation 




R129 


44 


54 


-10 


Interpreting a thermometer reading 




R128 


45 


55 


-10 


Sun is a star 




U619 


65 


77 


-12 






U626* 


58 


72 


-14 






R123* 


50 


66 


-16 


What scientists learn from fossils 


13 


R217* 


55 


48 


7 


Burning gasoline in a car creates heat 




R227* 


31 


25 


6 


Center of memory is cerebrum 




U718 


67 


61 


6 




U738 


45 


40 


5 






R234* 


53 


62 


-9 


Balance beam — weight in pah; weight on 












hook 




U736 


33 


44 


-9 




17 


U816* 


67 


60 


7 






U840* 


26 


20 


6 






U861* 


18 


29 


-11 




Mult 


U960 


15 


29 


-14 






U920* 


50 


64 


-14 






U924* 


39 


56 


-17 





♦Exercise also identified as atypical before balancing. 

Note: line separates atypical high from atypical lew exercises. 
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Median 



Exhibit 8-8 



Exercises showing atypical effects, balanced, 
for the Central for all ages 



% Correct 

Exer- % Dif- 

Age cise # Central Nat 1 ! ference Content 



9 


U635* 


69 


61 


8 






R123* 


73 


66 


7 


What scientists learn from fossils 




R129 


61 


54 


7 


Interpreting a thermcneter reading 




U640 


48 


53 


-5 






R126 


54 


59 


-5 


Center of Earth is hot 




R138* 


30 


35 


-5 


Clearing skies follow cold front 


13 


U758* 


44 


49 


-5 






U760* 


36 


42 


-6 




17 


U828* 


46 


39 


7 




Mult 


U960* 


44 


29 


15 






U962* 


32 


22 


10 





*Exercise also identified as atypical before balancing. 

Note: line separates atypical high fran atypical low exercises. 
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Of the 11 exercises identified as atypical and listed in 
Exhibit 8-8, five show a Central disadvantage and six a Central 

advantage . Eight were among the twelve exercises identified as 
atypical in the distributions of unadjusted effects in Report 4, 

and three appear as atypical here for the first time. Two of the 
four atypical exercises classified as Objective 2 show a positive 
effect and two a negative. Here, even more than in the case of 
the effects before balancing, the general impression is one of 
lack of pattern. Thus, we can only note that there are some 
large deviations from the median without proposing hypotheses 
to explain the deviations. 



West 

The distributions of balanced effects for the West are pictured 
in Exhibit 8-9 and the atypical exercises are listed in Exhibit 
8-10. The balancing has reduced somewhat the median effect# at 
ages 17 and Adult (from 1.7% and 2.6% before balancing to 0.7% 
and 1.9% after balancing) , and when these percentages are com- 
pared with the estimates of fluctuations to be expected from 
sample to sample, :t is no longer possible to say that the 
performance in the West is any better than that for the country 
as a whole. The medians at ages 9 and 13 were not reliably 
different from the national performance before balancing, and 
the same remark applies to the medians after balancing. 

Of the 13 exercises identified as atypical after balancing, 11 
were so identified before balancing. When the four exercises 
showing an atypical advantage for the West are compared with 

the nine showing an atypical disadvantage, we identified no 
meaningful contrasts in terms of content of exercises. 
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Exhibit 8-9 
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Distributions of West regional effects 
for all science exercises at four 



balanced, 

ages 



Age 




No. of 

Exercises 145 122 

Median 0.6% -0.8% 



124 

0.7% 



119 

1.9% 





Exhibit 8-10 



Exercises showing atypical effects, balanced, 



for the West for all ages 



% Correct 





Exer- 






% Dif- 




Age 


cise # 


West 


Nat'l 


ference 


Content 


9 


U640* 


63 


53 


10 






U647* 


43 


36 


7 






U667 7 


55 


63 


-8 






U669* 


44 


52 


-8 




13 


U718 


54 


61 


-7 






U711* 


68 


76 


-8 






U728* 


44 


53 


-9 




17 


U854* 


72 


64 


8 






U814* 


72 


65 


7 






U862 7 


19 


25 


-6 






U828* 


32 


39 


-7 




Adult 


R445 7 


1 


IT' 


-10 


Find density of vrood block losing ruler 












and balance 




U960 


17 


29 


-12 





*Exercise also identified as atypical before balancing. 

Note: line separates atypical high from atypical lew exercises. 
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Chapter 9 



SEX DIFFERENCES AFTER BALANCING 



Since the sexes are relatively well balanced across the other 
classifications, we do not anticipate and do not find any 
remarkable effects due to balancing. Exhibit 9-1 shows the 
medians for each age. Consequently, a discussion of the 
balanced sex differences must lead to the same general con- 
clusions as those reached from discussion of the unadjusted 
sex differences, as given in NAEP Report 4. Accordingly, no 
further comments are necessary here. 



Exhibit 9-1 



Median sex differences for four ages, 
unadjusted and balanced 



Median 


difference : 


Male minus 


female 


Age 


Unadjusted 


Balanced 


Change 


9 


1.8 


1.4 


0.4 


13 


2.9 


2.4 


0.5 


17 


5.0 


4.0 


1.0 


Adult 


9.9 


9.2 


0.7 
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APPENDIX A 



METHODS 



This report presents unadjusted and balanced results at four 
ages (9, 13, 17, and young Adults) classified in several ways-- 
unadjusted results: relative performance of Blacks relative to 

Non-Blacks, five different levels of parents’ education, and 
seven levels of size -and- type of community balanced results: 

Males and Females, residents of four different geographic regions 
of the nation as well as the three unadjusted classifications 
mentioned above.* Our concern is to describe how well these 
groups of respondents perform on science exercises. 

In the interest of clarity and ease, we want to make as few 
numerical comparisons as will adequately describe what we are 
to observe. When there are only two groups in a classification, 
as with sex divided into Male and Female, one numerical com- 
parison will suffice. We need only compare one group with the 
other for each exercise. For sex we will use the difference 



Sex % 

Difference = |f 

For example, one exercise given at age 9 shows 76% of the Males 
and 68% of the Females responding successfully. Using the 
.procedure above, 

Sex 

Difference = 76% - 68% = 8%. 



Success 
r Males 



f 



% Success "I 
For Females] 



Where there are more than two groups in a classification, such 
differences are not as simple to use. With four groups, such 
as four regions or four community sizes, simplicity is gained 
by comparing the percentage of success of respondents in each 
group with that in the nation as a whole. We shall call the 
number thus obtained an effect. One example would be 



Northeast 




["Northeast"! 




[National”] 


Effect 


= 


1% Success] 


- 


1% Success] 



* See Report 4 for unadjusted results for Male-Female and 
geographic region. 



For one exercise which 66% of 9-year-olds in the Northeast and 
55% of all 9-year-olds answered successfully, the regional 
effect is 

Northeast 

Effect = 66% - 55% = 11% 



What Should Be Described? 



For people of the same age, when we compare the performance of 
a group--in a geographic region or from a size of community-- 
with all people of that age in the country, the results are 
different for each exercise — different, but not too different. 
Even though some exercises show an advantage for the group, 
others a deficit, there is often a clear tendency for the exer- 
cises that we are considering as a whole to lean one way or 
the other. We describe this tendency by reporting the typical 
behavior of exercises. 



Since not all exercises show the same advantage or deficit for 
the group, description of the typical behavior of all exercises 
in the class is usually not enough. We can learn something 
more from studying individual exercises for which the group 
advantage or deficit differs substantially from this typical 
behavior. If we use an appropriate rule to identify these 
atypical exercises , it will be worthwhile to mention each of 
them individually, telling something of their nature and re- 
porting what advantage or deficit each shows when the group is 
compared with the whole couitry. This concentration on atypical 
exercises is particularly helpful because exercises showing close 
to typical behavior are often lacking in distinctive flavor. 
Atypical exercises, by contrast, often tell us something. 



When we deal with the two sexes, we do the same things, except 
we shall compare Males with Females, rather than a group with 
the whole country, both in terms of typical behavior and in 
terms of atypical exercises. 

Classes of Exercises 



Another approach to the discussion of our results is possible 
because the science exercises may be separated into classes by 
some common feature. We may investigate the behavior of the 
exercises for each class, and then ask how this behavior chances 
from one class to another. 



Only a few separations into classes are considered in this 
report. One of these is provided by content. Most exercises 
call on knowledge of either physical or biological science. 

Another classification is provided by the four science objectives . 

1. Know fundamental facts and principles of science; 

2. Possess the abilities and skills needed to engage in 
the processes of science; 

3. Understand the investigative nature of science; 

4. Have attitudes about and appreciations of scientists, 
science, and the consequences of science that stem 
from adequate understandings. 

Although other ways of classifying the exercises might be at 
least as useful in describing sex differences, regional effects, 
or community size effects, we have not found them. 

What Shall Be Typical? 

In the interest of simplicity and clarity, we choose to 
summarize the behavior of a set of exercises by the median (or 
midpoint) of the differences from national results. The median 
is a summary value such that an equal number of the values 
summarized lie on either side of it. For the 9-year-old sample 
— for instance, as in Exhibit 2-1, there were 145 exercises — 72 
exercises where the Black effect was greater than -14.5% and 72 
exercises below -14.5%. Because equal numbers were more and 
less than -14.5%, we call -14.5% the median effect and use it to 
summarize all 14 5 effects for the Black group. 

The median is an effective typical value because (1) it is 
easy to understand and (2) it is little affected by the presence 
or absence of unusual values. 



* Norris, Eleanor L. (Ed.) Science Objectives . Ann Arbor, 
Michigan: Committee on Assessing the Progress of Education, 

1969. 



What Shall be Atypical 



Only a sample of each age group responded to each exercise, 
yet National Assessment results are concerned with whole 
populations — sometimes with all 9-year-olds in the country, 
sometimes with all 13-year-old boys, sometimes with all 17- 
year-olds in the Northeast, sometimes with all Adults in small 
cities and so on. Thus there must always be a step from 
sample to population in interpreting National Assessment re- 
sults, allowing for the inescapable small differences between 
sample results and population facts. This inevitably raises 
statistical considerations — considerations which ought to guide 
our approaches, our exposition, and our statements without in- 
terfering excessively with the reporting and discussion of 
results . 

The samples taken in National Assessment were well designed 
scientific probability samples. As a result, it would be 
possible to assess how large the differences between sample 
value and population value are likely to have been for almost 
any quantity we chose to calculate from the results — for 
example, a Black effect. Results of such calculations have 
guided the choices in this report of how large a departure 
from typical (medium) behavior qualifies an exercise for 
separate consideration. Appendix C discusses considerations 
involved in, and the exact nature of, this choice. 

The chosen rule sets cutoffs determining which comparisons we 
call atypical . The cutoffs found by applying the chosen rule 
are shown by two arrows on each picture of the distribution of 
comparisons. Exercises corresponding to comparisons outside 
these cutoffs are discussed individually. 
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APPENDIX B 



BALANCING 



Purpose 

This is the first of the NAEP reports to look at more than 
one characteristic of respondents at a time. We try to 
reduce the extent to which effects of one characteristic 
can masquerade as effects of another characteristic. 

Proportionate Numbers Create No Problems 

We first look at some simple examples of what can — and, 
in appropriate circumstances, must — happen. Assume 
that we are assessing the effects of one characteristic 
that classifies respondents into groups A, B and C and of 
another characteristic that classifies respondents into 
either group U or V. The two characteristics combined to 
form six combination subgroups, AU, AV, BU, BV, CU, CV. 

For our first set of examples, we assume that the numbers of 
respondents in each of these six subgroups are as shown in 
the upper left corner of Exhibit B-l. (The first three 
examples all use this same distribution of supposed 
observations.) Note that, for every group on the first 
characteristic (A, B and C) , representation from group U 
stands in a fixed proportion to that from group V. Also 
note that we have proportionate numbers from groups 
A, B and C in both groups U and V. 

In example 1, every U subgroup — whether it is AU, BU or 
CU — has 60% success, and every V subgroup — whether it is 
AV, BV, CV — has 40% success. Since the AU subgroup is of 
size 100 and has 60% success, it includes 60 successes 
(60% of 100) . The numbers of successes for the other five 
subgroups are calculated similarly. To find the total number 
of successes for all of group U, we have only to add up the 
number of successes for each of the three subgroups in U, 
finding 60+120+180=360. Similarly, we add to find the 
total number of respondents in group U, 100+200+300=600. 

The unadjusted percentage of success in group U is then 
360/600=60%. When we do the same for group V and, as well, 
for groups A, B, C (defined by the other characteristic), 
we find very satisfactory results. Group U has a percentage 
of success that is 12% better than the overall percentage 
of success, 48%. Group V has a percentage of success that 
is 8% poorer than the overall percentage of success. No 
difference appears among groups A, B and C. Thus, in this 



Exhibit B-l 



Some examples of simple behavior when 



the numbers of cases are proportional 



Example 1 











% of 




# 


of 








# of 


cases 


success 




successes 






U 


V 








u 


V 






A 


100 


150 


250 


60% 40% 


A 


60 


60 


120 


48% 


B 


200 


300 


500 ' 


60% 40% 


B 


120 


120 


240 


48% 


C 


300 


45J 


750 


60% 40% 


C 


180 


180 


360 


48% 




600 


900 


1500 






360 


360 


720 
















60% 


40% 




48% 


Example 


2 
























% of 




# 


of 








# of 


cases 


success 




successes 






u 


V 








u 


V 






A 


100 


150 


250 


40% 40% 


A 


40 


60 


100 


40% 


B 


200 


300 


500 


50% 50% 


B 


100 


150 


250 


50% 


C 


300 


450 


750 


60% 60% 


C 


180 


270 


450 


60% 




600 


900 


1500 






320 


480 


450 
















53.3% 


53.3% 




53.3% 


Example 


3 
























% of 




# 


of 








# of 


cases 


success 




successes 






u 


V 








u 


V 






A 


100 


150 


250 


50% 30% 


A 


50 


45 


95 


38% 


B 


200 


300 


500 


60% 40% 


B 


120 


120 


240 


, 48% 


C 


300 


450 


750 


70% 50% 


C 


210 


225 


435 


58% 




600 


900 


1500 






380 


390 


770 
















63.3% 


43.3% 




51.3% 



1 . 0 '. 
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case, the percentages in the five marginal totals that 
correspond to simple unadjusted results are completely 
consistent with the percentages for the six subgroups 
that generate the data. 

Example 2 shows how differences between A, B and C, given 
proportionate numbers in the subgroups, will appear as of 
the proper size when computed as above. In this example, 
the difference between U and V is zero. 

Example 3 is a little more complex. It assumes that each 
U subgroup does 20% better than the corresponding V subgroup, 
but that there are also differences when we change from C to 
B or B to A. Again directly computing results from the 
groups (A, B, C, U, V) causes no trouble. The overall 
success is 51.3%. The percent of success for group A 
is 38%, that is, 13.3% below the total percent of success. 

The percent of success for group U is 63.3, that is, 12% 
above the total percent of success. And when we add together 
the three effects, we obtain the percent of success for the 
AU subgroup, i.e., 51.3% - 13.3% + 12.0% = 50%. The result 
of combining the effects for any of the other combinations 
is equally consistent. This consistency is something special; 
it depends essentially on the sizes of the six subgroups 
being proportional. 

Disproportionate Numbers Can Make Trouble 

We now turn to a different set of examples, where the 
numbers of respondents are not proportional within the 
categories. The upper left-hand corner of Exhibit B-2 
shows the new pattern. 

Example 4, shows the same pattern for percent of successes 
as Example 1 where there is no difference between A, B and C 
within levels U and V and equal differences between U and V 
at each level of A, B and C. . Thus, in this example 
the marginal totals accurately reflect the difference between 
the U and V subgroups but they give quite an erroneous 
picture of the A, B and C effects. By examining the 
distribution of the number of cases it is evident that the 
marginal (unadjusted) ratio of successes to cases for 
category C is in actuality measuring the effect of V. Thus, 
in this example, one variable masquerading for another has 
affected the unadjusted estimates of the effects of the 
A-B-C categories. This would not be apparent if we had 
not simultaneously looked at both the A-B-C and U-V 
classifications. Given disproportionate numbers, the true 
difference between U and V both appears as itself and 
masquerades as differences among A, B and C. 
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Some examples of simple behavior 
when the numbers are disproportionate 



Example 4 











% of 




# 


of 








# of 


cases 


success 




successes 






U 


V 








u 


V 






A 


100 


100 


200 


60% 40% 


A 


60 


40 


100 


50% 


B 


50 


150 


200 


60% 40% 


B 


30 


60 


90 


45% 


C 


0 


200 


200 


60% 40% 


C 


— 


80 


80 


40% 




150 


450 


600 






90 


180 


270 
















60% 


40% 




45% 


Example 5 
























% of 




# 


of 








# of 


cases 


success 




successes 






U 


V 








u 


V 






A 


100 


100 


200 


40% 40% 


A 


40 


40 


80 


40% 


B 


50 


150 


200 


50% 50% 


B 


25 


75 


100 


50% 


C 


0 


200 


200 


60% 60% 


C 


— 


120 


120 


60% 




150 


450 


600 






65 


235 


300 
















43.3% 


52.2% 




50% 


Example 6 
























% of 




# 


of 








# of 


cases 


success 




successes 






U 


V 








u 


V 






A 


100 


100 


200 


50% 30% 


A 


50 


30 


80 


40% 


B 


50 


150 


200 


60% 40% 


B 


30 


60 


90 


45% 


C 


0 


200 


200 


70% 50% 


C 


— 


100 


100 


50% 




150 


450 


600 






80 


190 


270 





53.3% 42.2% 4 5% 
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Example 5 shows how differences among A, B and C can 
masquerade as differences between U and V. Example 6 shows 
how each of two sets of real differences can both appear for 
themselves and masquerade as the other. 

In none of these examples do we obtain the percent of 
success for the subgroups by combining the overall effect 
and the unadjusted group effects as we did when the numbers 
of cases were proportional. Thus, in Example 6, the overall 
percent of success is 45, the A effect is 5% lower and the U 
effect is 8.3% higher , and when we add we get 45% - 5% + 8.3% 

= 48.3%, not 50% as we would like. However, if we assume 
an A effect of -10%, a B effect of 0%, a C effect of +10%, 
a U effect of +15% and a V effect of -5% , then the 
combinations are consistent. For example, 45% — 10% + 15% 

= 50% for the AB combination and 45% + 10% - 5% = 50% for the 
CV combination. It has been possible, then, to find a set 
of effects that has removed the distortion due to 
disproportionate numbers in the subgroups, at least in this 
instance where the group effects are the same from one 
subgroup to another. 

The Need 

Disproportionate numbers in real population groups produce 
the problem that we have been discussing. For example, 
a larger fraction of Blacks are found both in the Inner City 
and in Rural Areas. Larger fractions of Blacks also have 
low parental education. When we look at the whole group 
of Black children, some of the deficit shown by the group 
comes from effects characterizing Inner Cities or Rural 
Areas and some from the effects of lower parental education. 

We should do what we can to see through this sort of confusion . 
It was possible to adjust the data in Example 6 simply by 
inspecting the patterns of percentages for the six subgroups. 
These patterns, however, were based on hypothetical data. 

In real data there are often complex effects for combinations 
in the subgroups over and above the group effects. 

Furthermore, the hypothetical data of Example 6 involved 
only two characteristics. In real data there are often more 
than two characteristics involved, and the simple inspection 
technique that we used in finding the relationships in 
Example 6 will no longer work. There are, however, a number 
of ways to carry out computations that can help to reduce 
the distortion due to disproportionality . We next state 
the intended result of the calculation, then we say how we 
go about obtaining it. 
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A Balanced Fit 



We intend to find group effects (expressed in percentages) 
that, when combined by addition with each other and with the 
overall percentage of success, give fitted percentages of 
success that correspond with the actual data in one simple 

ways _ _ - - 

■ — if we choose any group by a single characteristic, 
say group A in Example 6, and if we use the fitted 
percentages and the actual numbers of cases to 
calculate the number of successes for each subgroup 
that involves group A, and if we then add these 
calculated numbers of successes, the total number 
of successes over all subgroups will be the same 
as the total actually observed in the data. 

Suppose^ for example, that we have 600 cases distributed 
as in the upper left of Exhibit B-3 and that the total number 
of successes (270) is distributed among groups A, B, C, U, 
and V as shown in the upper right of Exhibit B-3. The overall 
percentage of success is 270/600 or 45%. Now, if we let 
A, B, C, U and V stand for the five group effects (in 
percentages), our definition of balancing says that the fitted 
number of successes in the AB subgroup may br represented 
by 100(45% + A + U) , the fitted number of successes in the AV 
subgroup is 100 (45% + A + V) , and the number of fitted 
successes in both subgroups by the sum of these two. Moreover, 
the definition says that this sum must be equal to the number 
of successes actually observed, which is 80. Thus, we can 
write the equation 100 (45% + A + U) + 100 (45% + A + V) =80. 

We can also write four other equations, two for the rows 
involving groups B and C and two for the columns involving 
groups A and U, as illustrated in the middle section of 
Exhibit B-3. If we try to solve these five equations for the 
five effects, however, we find that we are not able to obtain 
a unique solution. The equations are not independent since 
the three equations for the rows must sum to the overall 
total of 270 and likewise the two equations for the columns. 

We have only three independent equations, two for the rows 
and one for the columns, and we need two additional independent 
equations in order to find a single solution. 

There is an additional requirement established by our 
definition. Overall, the effects of A, B and C must balance 
each other since all that is left is the overall effect; 
likewise, the effects of U and V must balance each other. 

This means that we can write two additional equations based 
on the numbers of cases in the margins of our table. These 
are: 
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Exhibit B-3 



An example of the procedure used in 
obtaining a balanced fit 



Example 7 



# of 

# of cases succeses 





U 


V 






U 


V 




A 


100 


100 


200 


A 






80 


B 


50 


150 


200 


B 






90 


C 


0 


200 


200 


C 






100 




150 


450 


600 




80 


190 


270 



The representation of the fitted successes in the sub-groups 



i-group 


Repr e sentation 


AU 


100(45% + A + U) 


AV 


100(45% + A + V) 


BU 


50(45% + B + U) 


BV 


150(45% + B + V) 


CU 


0 


CV 


200(45% + C + V) 



Equations 



Combination 

AU + AV 
BU + BV 
CU + CV 
AU + BU + CU 
AV + BV + CV 
A + B + C 
U + V 



100(45% + A + U) + 100(45% + A + V) = 80 

50(45% + B + U) + 150(45% + B + V) =90 

200(45% + C + V) =100 
100(45% + A + U) + 50(45% + B + U) + 0 = 80 

100(45% + A + V) + 150(45% + B + V) + 200(45% + C + V) = 190 

200 (A + B + C) = 0 

150U + 450V = 0 



The solution 



A effect = - 10% 

B effect = 0% 

C effect = + 10% 

U effect = + 15% 

V effect = - 5% 



O 
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and 



150 (45% + U) + 450 (45% + V) = 270 

200 (45% + A) + 200 (45% + B) + 200 (45% + C) = 270 
and these reduce to 
150V + 450V = 0 

and 

20 0A + 200B + 200C =0 

If we then use these two last equations along with any two of 
the three equations based on the rows and any one of the two 
equations based on the columns , we have five independent 
equations and five unknowns that can be solved for the five 
unique balanced effects. The solution gives the values in 
the lower part of Exhibit B-3 , and it is observed that 
the solution is the same as we obtained in Example 6. 

It is easy to look at this example and say, "Ah, I see. 

This example is the same as Example 6 , and the balanced fit 
reproduces the number of successes in each subgroup 
exactly" . Notice, however, that in Example 7 there are no 
numbers of successes given for the subgroups. It is true 
that the balanced fit of Example 7 does produce an exactly 
matching fit to the actual data of Example 6, but it is 
also a balanced fit to data that have different frequencies 
of success in the subgroups. Exhibit B-4 shows several 
patterns of successes all having the same balanced fit. 

Balancing does not remove all the complications from the 
data. It does strip off the concealment involved in 
masquerades caused by disproportionate numbers. 

More General Cases 



We have considered only two characteristics, one with 
two groups, the other with three. The same technique 
extends to more groups. Extensions to more characteristics 
involve more complicated equations, but the interpretations 
are equally straightforward, so long as we realize that what 
we balance are the number — or equivalently the percents — 
of success for all groups defined by any single characteristic 
(and not necessarily for any subgroup defined by two or 
more characteristics) . 

• If, as in this report, we balance 

— sex, two groups 

— color, three groups (including unascertained) 

— region, four groups 

— education, five groups (including unascertained) 

— size and type of community, seven groups 

we have taken a long step to reduce masquerading. But it 
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Exhibit B-4 



Examples of possible observations 
leading to the same balanced fits 



The pattern of number of cases (for all examples) 



U V 



A 


100 


100 


200 


B 


50 


150 


200 


C 


0 


200 


200 




150 


450 


600 



Some alternative patterns of numbers of successes 





U 


V 






U 


V 




A 


50 


30 


80 


A 


40 


40 


80 


B 


30 


60 


90 


B 


40 


50 


90 


C 


“ 


100 


100 


C 


— 


100 


100 




80 


190 


270 




80 


190 


270 




U 


V 






U 


V 




A 


70 


10 


80 


A 


80 


0 


80 


B 


10 


80 


90 


B 


— 


90 


90 


C 


— — 


100 


270 


C 


— 


100 


100 




80 


190 


270 




80 


190 


270 



Remark 



Since the numbers of successes for the A, B, C, U, V single- 
characteristic groups are 80, 90, 100, 80, 190 for each of 
whe four examples, the fit of Exhibit B-3 is a balanced fit 
for each pattern. 
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would be wrong to think that we have gone the whole way, for 

— we have not yet used all, merely most, of the 
characteristics that were collected 

— we have only used these characteristics in a rather 
limited way (we could have used more than four 
regions, more types of communities, f inal__classifi- 
cations of years of school completed, for instance) 

— we know that the overt characteristics are less 
than perfect measures of the variables that should 
concern us (years of school is a less-than-perfect 
measure of parental education, which is itself a 
less-than-perfect measure of either home attitude 
toward education or available aid in homework) 

— there are other characteristics of importance, e.g., 
economic status, that were not measured. 

The deficiencies of balancing are clear; it cannot be the 
final answer. But the step from unadjusted comparisons to 
balanced ones is a long step from outward appearance 
toward — toward, not to — inward realities. The problem 
facing the schools is usually better shown by unadjusted 
values; the effectiveness of the educational process for a 
group is often better indicated by balanced results. 



APPENDIX C 



CHOOSING THE CUTOFFS 



Our observed percentages of success are based on samples, but 
our interest is in whole populations. Thus our information 
is always less than complete. As a result, conclusions 
stated either in numbers or in words are approximate rather 
than exact, slightly blurred rather than sharp. Whatever 
rule we choose to fix the cutoffs, and thus to decide which 
comparisons are atypical and worth serious consideration, 
the performance of that rule will be less than perfect. 

However we choose a cutoff rule, there will be some chance 
that an exercise whose population value is typical or close 
to typical will have a sample value classed as atypical by 
our rule. The more one tries to avoid this by stiffening 
the rule, thus calling fewer exercises atypical, the more 
often sample values will fail to be recognized as atypical 
when their population values are far from typical. This is 
a general problem that arises whenever sample results 
(including differences and effects) are examined in detail. 

In choosing a rule, we must take account of these two kinds 
of unfortunate consequences. We may also need to consider 
what sizes of differences or effects are likely to be 
practically meaningful. Besides discussing individual 
exercises, we want to discuss the groups of exercises found 
atypical in a given direction. Our choice of cutoffs 
affects our ability to descrive such groups. The rule 
whose details are given below was chosen with some 
attention to all these considerations. 

Variances and Standard Errors 



The samples drawn for the first assessment, both in and out 
of school, involved careful designs in which the country was 
divided into areas called strata. For each exercise, results 
were obtained for two groups of respondents in each stratum.* 



* The two groups come from two "primary sampling units" in 
each stratum. (See Appendix C of NAEP Report No. 1.) Because 
of the existence of planned half strata and the loss of some 
primary sampling units, losses that were duly allowed for in 
the estimation procedure, it was necessary to "collapse" a few 
pairs of strata by combining two original strata in one stratum 
for the purpose of assessing sampling variability. 
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To assess sampling variability, it is necessary to estimate 
a suitable measure of how far the results of the 1969-70 
administration of a science exercise would have varied had 
different groups of respondents been selected in any or all 
strata. Our concern has to be with other possible selections 
according to the same random procedures used in the actual 
administration. — - ..... 

In this situation, the use of two groups of respondents per 
stratum allows us to calculate an estimate of such a measure 
of variability, the sampling variance, defined as the 
average squard of the difference in result between a single 
administration and the average result for all possible 
admin i s trations . 

In this report, our immediate concern is with the estimated 
sampling variance of sex differences, regional effects, and 
size-of-community effects. It is equally useful to look at 
the standard errors of these comparisons, defined as the 
squard roots of the estimated sampling variances. 

These estimated sampling variances or standard errors will 
vary from one exercise to another for several reasons, 
including : 

1. Different sample sizes for some exercises. (The 
time-consuming apparatus exercises were given 

to fewer respondents. Certain other exercises 
were excluded in a few jurisdictions. Shortness 
of class periods occasionally caused omission 
of later exercises.) 

2. The automatic decrease in variance accompanying 
very easy or very hard exercises. 

3. Larger or smaller differences in the sizes of 
actual school-to-school differences. (In some 
cases these presumably reflect the greater or 
lesser effect or extent of curriculum variations.) 

4. Sampling variations inevitable in the process 
of estimating sampling variability. 

For reference. Appendix E presents estimated standard errors 
for effects, both unadjusted and adjusted, for each science 
exercise classified by objective. 



Kinds of Cutoffs 



If we wanted to look at results only for individual exercises, 
it would be natural to compare each deviation (whether from 
zero or from a typical comparison) with a suitable multiple 
of the standard error available for that deviation. Doing 
this would correspond, in terms of cutoffs, to using a 
separate cutoff for each exercise". This would have been 
quite possible by using the estimated standard errors given 
in Appendix E. However, it was felt that this would involve 
more details and complications than would be warranted by 
the likely gains. (The interested reader can use Appendix E 
as a basis for his own experiments with this type of 
analysis. ) 

Instead, the choice was to use a single pair of cutoffs for 
each class of exercises considered in the report. To do 
this, it was necessary to combine standard errors or estimated 
sampling variances across exercises. For this purpose, 
it is appropriate to use a combined standard error, which 
is the square root of a combined estimated sampling 
variance, this latter being the arithmetic mean of the 
estimated sampling variances for the exercises involved. 

Size of Cutoffs 



The easiest way to use the combined standard error to fix 
the cutoffs is to start from the corresponding median and 
lay off a suitable multiple of the combined standard error 
on either side. If the multiple is large, very few exercises 
will be falsely called atypical, but we will fail to take 
advantage of the information provided by intermediate 
deviations. 

There are circumstances under which we might be able to 
make good use of quite small multipliers, even fractions. 

When comparing one subclass of exercises well below the 
median with another subclass of exercises well above 
the median, larger subclasses could be helpful, so helpful 
as to outweigh the fact that a substantial fraction of 
each subclass would fall in that subclass purely by 
accident. 

A multiplier of about 2 is conventional for many purposes. 
This choice leads to about 5% of all items entering one 
distinctive subclass or the other purely by chance. 

Since we deal with classes of about 120 exercises (and 
since 5% of 120 is six) , this would mean an average of 
about 3 "atypically high" exercises and about 3 "atypically 
low" exercises purely by chance. Thus this choice seemed 
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more likely to confuse the picture than to clarify it. 

Another practical detail should concern us. Our exhibits 
show only the integer percentage for a difference or effect. 
Thus all comparisons between 3.0 and 3.9 are shown as 3, 
for example, if our cutoffs are to be simply related to 
our pictures, they should fal3 at the end of such a range. 

(A cutof f between 2.9 and 3.0 or one between 3 .9 and 4~. 0 

is easily pictured and used. One between 3.4 and 3.5, 
or at 3.74, is not.) 

As a result, it was decided to set the cutoffs in the 
following way: 

1. start from the median comparison for all the 
exercises involved, 

2. move up and down distances equal to 2.5 times 
the combined standard error, 

3. move further in each direction until the cutoff 
divides one integer-percent range from the next. 

Thus if the median were -1.7% and the combined standard 
error were 1.9%, we would find first 
(2.5) 1. 9%) = 4.75% 

then 

-1.7% - 4.75% = -6.45% and -1.7% + 4.75% = 3.05% 
and then place the arrows between the intervals 
-7.9 to -7 . 0 and -6.9 to -6.0 
on the low side and between the intervals 
3.0 to 3.9 and 4.0 to 4.9 

on the high side (i.e., locate the whole percentage interval 
containing the computed cutoff point, then place the cutoff 
arrows at the point dividing that interval from the adjusted 
interval furthest from the median.) 

For each class of rather more than 100 exercises, an average 
of perhaps one or two exercises can be expected to fall 
outside these cutoffs because of sampling fluctuations. 

(The smaller sample sizes for apparatus exercises can increase 
these numbers somewhat.) A substantial number of exercises 
that we might have liked to consider atypical will of 
course fail to be recognized as such. Some such compromise 
is, however, necessary. 
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DEFINITION OF TERMS 



x The major reporting categories used in this report are: 

~\ • Age, Region, Size and -Type of Community (STOC) , Sex , 

Color, and Parents' Education. 

1. Age: Four age levels were assessed. Three of these 

were in-school — 9,13, and 17-year-olds; and two — 17s 
and young Adults — were out-of-school, sometimes 
referred to as the household sample. The criteria 
used or the operational definitions of the four 
ages are as follows: 

9s — Born between 1/1/60 and 12/31/6 0 
13s — Born between 1/1/56 and 12/31/5 6 
17s — In-school: Born between 10/1/51 and 9/30/52 

Out-of-school: Born between 10/1/50 and 

9/30/51 and not enrolled in school in March '68 
OR born between 10/1/51 and 9/3 0/52 and not 
enrolled in school in March 1969. 
young Adults — Born between 7/1/33 and 6/30/43 

2. Region: Four geographical regions are being used for all 

age levels — Northeast , Southeast , Central and West . 

The states falling in each of these four regions, for 
reporting purposes, are as follows. 



Northeast 


Southeast 


Central 


West 


Del. 


Ala. 


N.D. 


Hawaii 


Maine 


Ark. 


S.D. 


Alaska 


N.H. 


Fla. 


Iowa 


Idaho 


Vt. 


Ga . 


Kansas 


Mont. 


D.C. 


Ky. 


Minn. 


Wyo . 


Md. 


La. 


Mo. 


Utah 


N.J. 


Miss. 


Neb. 


N.M. 


N.Y. 


N.C. 


111. 


Nev. 


Conn. 


S.C. 


Ind. 


Cal. 


Pa. 


Tenn . 


Mich. 


Ore . 


Mass . 


Va. 


Ohio 


Wash. 


R.I. 


W. Va. 


Wis. 


Ariz . 



Colo. 
Tex . 
Okla. 
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This classification is that used by the Office of Business 
Economics, Department of Commerce? the names for regions 
used by the OBE differ from National Assessment names for 
three regions: 

National Assessment QBE 

- -- Northeast — Northern Atlantic 

Southeast Southeast 

Central Great Lakes and Plains 

West West and Northwest 

3. Size and Type of Community (STOC) : Previous National 

Assessment reports have given results for four sizes of 
community (SOC) : Big Cities, Urban Fringe, Medium Size 

Cities, and Smaller Places. These are defined in 
National Assessment. Reports 4, 5, and 6. This (Report 7) 
and subsequent reports integrate three "extreme types 
of community," each composed of approximately 10% of 
the population, with four sizes of community representing 
the remaining 70% of the population into a seven-category 
STOC classification. 

NOTE: The four "size of community" categories within 

the . STOC classification are not equivalent to the four 
"size of community" categories within the SOC classification 
used in previous reports since the latter did not have 
the three "extreme types of community" extracted 
from them. 

The definitions of the extreme types of community 
were derived from an occupation question for both 
the in-school and out-of -school samples (see Exhibit D-l) . 

By classifying schools rather than counties, it was 
possible to identify much more uniform groups. One 
extreme group was selected in each of the three 
directions indicated by the exploratory analysis: 

(1) schools where high proportions were factory or 
professional workers, (2) city schools where a high 
proportion of parents were either not regularly 
employed or on welfare, and only a low proportion 
were professional or managerial, (3) near-city and 
city schools where a high proportion of parents were 
professional or managerial and only low proportions 
were factory or farm workers, not regularly employed 
or on relief. 

Smaller extreme groups would have been more extreme; 
larger extreme groups would have had better determined 




D-2 




* 









T3 
















in 


0 ) 
















0 ) 


M 




T3 












•H 


T3 




H 












M 


G (A 




0 












0 


•H M 


X 


■XI 










(0 


O' 


* 0 


0 ) 


0 ) 










0) 


0 ) 


P 


M 


(0 










•H 


P 


'd a) 


X 


G 










VI 


id 


G *H 


G 


0 










0 


o — 


03 M 


•H 


X 












i-t 


O 4 






w G 








0) 


P id 


* 0 




0 ) 


M 0 ) 








P 


h jd 


H P 


d 


p 


0 ) g 








id 


o a 


03 Qi 


G 


03 


O' <D 








o 


V Id 


0 


03 


■a > 


03 p 










Id g 


•H * 


d 


0 ) -H 


G 0 








g 




G w 


0 ) 


P P 


id P 








0 


X to 


X H 


P G 


TJ & 


g 








-H 


g a 


0 03 


ro 0 ) 


G 


T3 








P 


id 


<1)-H 


G g 


•H * 


g G 








id 


E 


P 0 ^ 


•H 0 ) 


M w 


V* 03 








04 


H O 


•H g 


* M 


M (A 


id 








0 


0 M 


*P P 


w 0 


*0 <D P 


P (A 








o 


0 Mh 


H P id 


nd pp 


G * Q) 


M 




TJ 




o 


Xl'-' 


03 0 P 


G 0) 


(d MM M 


<0 0 ) 




0 ) 




o 


0 


G 


03* * 


o a) o 


G M 


T3 


•H 






in 


0 *P 


P G 


0 ) & X X 


id o 


0 ) 


P 




■H 


i 


•rl (/) P 4 


HOfl) 


> p id 


X 


>1 


•H 




o 


P 


(A p <D 


id S g 


•H 0 ) 0 H 


(A 03 


0 


(A 




o 


0 


w a) o 


o w 


P o 


P H 


•H 


(A 




XI 


1 


0 ) O' X 


•H (A P 


Id -H T3 p 


Q) 


04 


Id 




a 


p 


P 03 0) 


P (DP 


M>d<D 


g e 


g 


rH 


r— 1 


CO 


0 


OC^ 


(D h id 


0) P 03 X 


P M 


<1> 


o 


1 


i 


o 


M 03 


H 03 M 


04 0 ) P 


Id Id 


G 


G 


Q 


P 

n 




& S 


uwu 


O CO o 


04 04 


D 


D 


P 


u 

1 
















•H 


P 
















XI 


2 
















•H 


O 
















X 


















X 


X) 


0 ) 














w 


0 


*3 












O 




id 


o 


< 


W 


CJ 


Q 


W 04 


O 




i 


o 
















a 


















*rl 


















a 


















(U 


















<u 








P 
















% 


id 










p 


<D 


rH 


H 


rH 










<D 


M 


03 


id 


rH 










XJ 


•H 


•H 


o 


O 












03 


P 


•H 


O 










a> 


G 


0 ) 


G 






nd 






o 


G 


O' 


X 


0) 




0 ) 






c 


0 


03 


o w 


0 




>i 






0) 


•H 


G 


0) p 


rH 




o 






X) 


P in 


03 


P 0) 


XX 




rH 






c 


in a) 


g 


* 






Ou 






o 


0 *H 




* p 


H 




g 






cu 


0 M 


M 


•H 0 


0 ) 




( 1 ) 






CO 


0 * 0 


0 


id 5 


X 










a ) 


tn 




o 


P 




>1 






u 


(A 0) 


H 


•h 


o 


(A 


rH 






u 


- P 


id h 


P 0) 




p 


P 






o 


h id 


G 0 ) 


0) H 


M 


a) 


id a) 






u 


03 O 


0 G 


rH H 


O (0 


M 


rH P 








PU 


•H G 


O -H 


p 


M 


d id 








•rl 


U) 0 


* 


>i 0) 


0 










0 


m m 


* IA 


M * 


S 


0) rH 








G 


0 ) M 


CO 


O P 




p d) 








•rl 


P 0) 


<D M 


P 0 


g 


& 








M 


0 04 


H 0 


u s 


P 


P 








04 


u 


id 


id 


id 


o c 










04 


CO 


ip 


&4 


13 O 





ERIC 



percentages of success. The sizes of the three extreme 
groups, close to 10% of all those assessed, were chosen 
as a compromise between more extremeness and better 
determination. (Capital letters, below, represent 
codes for categories in Exhibit D-l.) 

STOC 1. Extreme Rural : D - (C + 2A) in %. This 

category represents 9.6% of all respondents 
extracted from three of the four old SOC 
categories as follows: 

Urban Fringe: 0.4% — Population 

less than 3,500 and all counties near 
a big city within the same Standard 
Metropolitan Statistical Area (SMSA) . 

Medium Size City: 1.7% — Population 

less than 3,500 and all counties within 
the same SMSA as a Medium Size City 
and not in a Big City county. 

Smaller Places: 7.5% — Population 

less than 3,500. 

STOC 2. Extreme Inner City : (E + F) - A in % . This 

category represents 9.9% of all respondents 
extracted from three of the four old SOC 
categories as follows: 

Big City: 8.5% — Population greater 

than 200,000 and low Socio-Economic 
Status (SES) within the city limits. 

Urban Fringe: 0.5% — Population 

greater than 150,000 and low SES 
near a big city and within the same 
SMSA. 

Medium Size City: 0.9% — Population 

from 150,000 to 200,000 and low SES 
within the same SMSA as a city with a 
population from 150,000 to 200,000. 

STOC 3. Extreme Affluent Suburb : A - (C+D + E + F) 

in % . This category represents 10.0% of all 
respondents extracted from three of the four 
old SOC categories as follows: 

Big City: 2.9% — Population greater 

than 200,000 and high SES within the 
city limits. 

Urban Fringe: 5.4% — Population greater 

than 150,000 and high SES near a big city 
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and within the same SMSA. 

Medium Size City: 1.7% — Population 

from 150,000 to 200,000 and high SES 
within the same SMSA as a city with a 
population from 150,000 to 200,000. 



STOC 4. Inner City Fringe : This category represents 



11.8% of all respondents and replaces the old 
Big City category. Population greater than 
200,000 and middle SES within the city limits. 



STOC 5. Urban Fringe : This category represents 16.7% 



of all respondents and replaces the old Urban 
Fringe category. Population from 15 0,000 
to 200,000 and middle SES; population from 
3,500 to 150,000 and all SES's. 



STOC 6. Medium-Size City : This category represents 21.4% 



of all respondents and replaces the old 
Medium-Size City category. Population 
150,000 to 200,000 and middle SES; and all 
SES's in the same SMSA containing a city 
with a population from 25,000 to 200,000. 



STOC 7. Small City : This category represents 20.7% of 



all respondents and replaces the old Smaller 
Places category. Population between 3,500 
and 200,000 and all SES's. 



4. Sex: Data are reported for Males and Females. 

5. Color: Data are reported for Blacks and Non-Blacks. 

Color classification was based on observation of the 
respondent at the time of the administration by the 
field staff member who administered the package. 

6. Parents' Education: All data are reported in terms of 

the highest level of education of either parent of 
guardian of a 17-year-old respondent. Parents ' 
Education is reported as follows: 

Level 1. Less than or equal to 8th grade 
Level 2. More than 8th grade but not a high 



school graduate 

Graduated from high school 

Some formal education beyond high school 




Level 3 . 
Level 4 . 



APPENDIX E 



NATIONAL PERCENTAGES OF SUCCESS, EFFECTS AND STANDARD 
ERRORS FOR REGION, SEX, SIZE AND TYPE OF COMMUNITY 
(STOC) , COLOR AND PARENTS' EDUCATION— UNADJUSTED 
AND BALANCED — FOR RELEASED AND UNRELEASED. 
EXERCISES BY OBJECTIVE 



Data on the percentage of success, effects, and standard errors 
of effects (unadjusted and balanced) for each Science exercise 
are shown for the four ages — 9, 13, 17 and Adult. Within each 
age group they are identified as released or unreleased ex- 
ercises by a three-digit number preceded by an R (released) or 
a U (unreleased) as follows: 



Age 


Released 


Unreleased 


9 


RlOO's 


U600 ' s 


13 


R200 ' s 


U700 ' s 


17 


R300 ' s 


U8 00 ' s 


Adult 


R400 ' s 


U900 ' s 



(All released exercises are reproduced in NAEP Report 1.) 

The exercises are listed acording to Objective 1 first, 
followed in turn by exercises for Objectives 2, 3 and 4. For 
each objective, the released (unreleased) exercise numbers are 
assigned in order from highest to lowest percentage of National 
success. Thus, within each age the exercises are divided into 
two main groups — released and unreleased — with the exercises 
listed in order of decreasing National success within each 
objective . 

In this appendix, the first line corresponding to each exercise 
shows the identifying number, National percentage of success, 
and for released exercises, a short description of the exercise 
content. The next four lines show the unadjusted effects and 
standard errors and the balanced effects and standard errors, 
respectively, for the five classifications of respondents in- 
cluded in this report (Region, Sex, STOC, Color and Parental 
Education) . Exercises which overlap other exercises at other 
ages are listed in Appendix C of Report 4. Identification of 
physical and biological science classifications are also in- 
cluded in Appendix C of Report 4 . 
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?TFo. »i22 *»TL % 71-i Honeybees are more beneficial to man than are lice, locusts, or termites. 
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EXER: R 1 35 WATL % 38.2 A rock broken into three pieces raises the water level in a container the same amoant as the whole rock 
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MIR: R 22 B RiTL % 25,9 Th« pttstnce of an ocehn fish fossil on a Mountain oatcrop is best explained by the hypothesis that the Mountain was 

raise A ap after the fish -had died. 
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HIM: *3*5 1ATL * 39.8 tHllze that atoms are rarely destroyed so that carbon atoms in a piece of bread could hate been part of a dinosaur* 
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